## File: README.md [](https://github.com/osm-search/Nominatim/actions?query=workflow%3A%22CI+Tests%22) Nominatim ========= Nominatim (from the Latin, 'by name') is a tool to search OpenStreetMap data by name and address (geocoding) and to generate synthetic addresses of OSM points (reverse geocoding). An instance with up-to-date data can be found at https://nominatim.openstreetmap.org. Nominatim is also used as one of the sources for the Search box on the OpenStreetMap home page. Documentation ============= The documentation of the latest development version is in the `docs/` subdirectory. A HTML version can be found at https://nominatim.org/release-docs/develop/ . Installation ============ The latest stable release can be downloaded from https://nominatim.org. There you can also find [installation instructions for the release](https://nominatim.org/release-docs/latest/admin/Installation), as well as an extensive [Troubleshooting/FAQ section](https://nominatim.org/release-docs/latest/admin/Faq/). [Detailed installation instructions for current master](https://nominatim.org/release-docs/develop/admin/Installation) can be found at nominatim.org as well. A quick summary of the necessary steps: 1. Clone this git repository and download the country grid git clone https://github.com/osm-search/Nominatim.git wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz 2. Create a Python virtualenv and install the packages: python3 -m venv nominatim-venv ./nominatim-venv/bin/pip install packaging/nominatim-{api,db} 3. Create a project directory, get OSM data and import: mkdir nominatim-project cd nominatim-project ../nominatim-venv/bin/nominatim import --osm-file 2>&1 | tee setup.log 4. Start the webserver: ./nominatim-venv/bin/pip install uvicorn falcon ../nominatim-venv/bin/nominatim serve License ======= The Python source code is available under a GPL license version 3 or later. The Lua configuration files for osm2pgsql are released under the Apache License, Version 2.0. All other files are under a GPLv2 license. Contributing ============ Contributions, bug reports and pull requests are welcome. When reporting a bug, please use one of the [issue templates](https://github.com/osm-search/Nominatim/issues/new/choose) and make sure to provide all the information requested. If you are not sure if you have really found a bug, please ask for help in the forums first (see 'Questions' below). For details on contributing, have a look at the [contribution guide](CONTRIBUTING.md). Questions and help ================== If you have questions about search results and the OpenStreetMap data used in the search, use the [OSM Forum](https://community.openstreetmap.org/). For questions, community help and discussions around the software and your own installation of Nominatim, use the [Github discussions forum](https://github.com/osm-search/Nominatim/discussions). --- ## File: docs/develop/data-sources.md # Additional Data Sources This guide explains how data sources other than OpenStreetMap mentioned in the install instructions got obtained and converted. ## Country grid Nominatim uses pre-generated country borders data. In case one imports only a subset of a country. And to assign each place a partition. Nominatim database tables are split into partitions for performance. More details in [osm-search/country-grid-data](https://github.com/osm-search/country-grid-data). ## US Census TIGER For the United States you can choose to import additional street-level data. The data isn't mixed into OSM data but queried as fallback when no OSM result can be found. More details in [osm-search/TIGER-data](https://github.com/osm-search/TIGER-data). ## GB postcodes For Great Britain you can choose to import Royalmail postcode centroids. More details in [osm-search/gb-postcode-data](https://github.com/osm-search/gb-postcode-data). ## Wikipedia & Wikidata rankings Nominatim can import "importance" data of place names. This greatly improves ranking of results. More details in [osm-search/wikipedia-wikidata](https://github.com/osm-search/wikipedia-wikidata). --- ## File: docs/develop/Database-Layout.md # Database Layout ### Import tables OSM data is initially imported using [osm2pgsql](https://osm2pgsql.org). Nominatim uses a custom flex style to create the initial import tables. The import process creates the following tables: The `planet_osm_*` tables are the usual backing tables for OSM data. Note that Nominatim uses them to look up special relations and to find nodes on ways. Apart from those the osm2pgsql import produces three tables as output. The **place_postcode** table collects postcode information that is not already present on an object in the place table. That is for one thing [postcode area relations](https://wiki.openstreetmap.org/wiki/Tag:boundary%3Dpostal_code) and for another objects with a postcode tag but no other tagging that qualifies it for inclusion into the geocoding database. The table has the following fields: * `osm_type` - kind of OSM object (**N** - node, **W** - way, **R** - relation) * `osm_id` - original OSM ID * `postcode` - postcode as extacted from the `postcal_code` tag * `country_code` - computed country code for this postcode. This field functions as a cache and is only computed when the table is used for the computation of the final postcodes. * `centroid` - centroid of the object * `geometry` - the full geometry of the area for postcode areas only The **place_interpolation** table holds all [address interpolation lines](https://wiki.openstreetmap.org/wiki/Addresses#Interpolation) and has the following fields: * `osm_id` - original OSM ID * `type` - type of interpolation as extracted from the `addr:interpolation` tag * `address` - any other `addr:*` tags * `nodes` - list of OSM nodes contained in this interpolation, needed to compute the involved housenumbers later * `geometry` - the linestring for the interpolation (in WSG84) The **place** table holds all other OSM object that are interesting and has the following fields: * `osm_type` - kind of OSM object (**N** - node, **W** - way, **R** - relation) * `osm_id` - original OSM ID * `class` - key of principal tag defining the object type * `type` - value of principal tag defining the object type * `name` - collection of tags that contain a name or reference * `admin_level` - numerical value of the tagged administrative level * `address` - collection of tags defining the address of an object * `extratags` - collection of additional interesting tags that are not directly relevant for searching * `geometry` - geometry of the object (in WGS84) A single OSM object may appear multiple times in this table when it is tagged with multiple tags that may constitute a principal tag. Take for example a motorway bridge. In OSM, this would be a way which is tagged with `highway=motorway` and `bridge=yes`. This way would appear in the `place` table once with `class` of `highway` and once with a `class` of `bridge`. Thus the *unique key* for `place` is (`osm_type`, `osm_id`, `class`). How raw OSM tags are mapped to the columns in the place table is to a certain degree configurable. See [Customizing Import Styles](../customize/Import-Styles.md) for more information. ### Search tables The following tables carry all information needed to do the search: The **placex** table is the central table that saves all information about the searchable places in Nominatim. In simpler terms, the `placex` table can be seen as the final, processed version of OSM data that is ready for search. While the `place` table contains raw imported data, `placex` stores enriched and indexed data that includes ranking, hierarchy (parent-child relationships), and computed metadata such as importance and postcode. Most search queries in Nominatim ultimately read from this table, making it the core table for forward and reverse geocoding. The basic columns are the same as for the place table and have the same meaning. The placex tables adds the following additional columns: * `place_id` - the internal unique ID to identify the place * `partition` - the id to use with partitioned tables (see below) * `geometry_sector` - a location hash used for geographically close ordering * `parent_place_id` - the next higher place in the address hierarchy, only relevant for POI-type places (with rank 30) * `linked_place_id` - place ID of the place this object has been merged with. When this ID is set, then the place is invisible for search. * `importance` - measure how well known the place is * `rank_search`, `rank_address` - search and address rank (see [Customizing ranking](../customize/Ranking.md) * `wikipedia` - the wikipedia page used for computing the importance of the place * `country_code` - the country the place is located in * `housenumber` - normalized housenumber, if the place has one * `postcode` - computed postcode for the place * `indexed_status` - processing status of the place (0 - ready, 1 - freshly inserted, 2 - needs updating, 100 - needs deletion) * `indexed_date` - timestamp when the place was processed last * `centroid` - a point feature for the place * `token_info` - a dummy field used to inject information from the tokenizer into the indexing process For implementation details, see the SQL definition in `lib-sql/tables/placex.sql` and the SQLAlchemy schema in `src/nominatim_api/sql/sqlalchemy_schema.py`. The **location_property_osmline** table is a special table for [address interpolations](https://wiki.openstreetmap.org/wiki/Addresses#Using_interpolation). The columns have the same meaning and use as the columns with the same name in the placex table. Only the following columns are special: * `startnumber`, `endnumber` and `step` - beginning and end of the number range for the interpolation and the increment steps * `type` - a string to indicate the interval between the numbers as imported from the OSM `addr:interpolation` tag; valid values are `odd`, `even`, `all` or a single digit number; interpolations with other values are silently dropped Address interpolations are always ways in OSM, which is why there is no column `osm_type`. The **location_postcodes** table holds computed postcode assembled from the postcode information available in OSM. When a postcode has a postcode area relation, or when the postcode geometry is [imported via JSONL files](../customize/Postcodes.md#jsonl-format) then the table stores its full geometry. For all other postcodes the centroid is computed using the position of all OSM objects that reference the same postcode. The `osm_id` and `is_area` fields can be used to distinguish the two. When `osm_id` is set, it refers to the OSM relation with the postcode area, and `is_area` is `true` for postcodes with a mature geometry (either from a postcode OSM area relation or imported via JSONL), `false` for postcodes without a mature geometry (guessed postcode geometries). The meaning of other columns in the table is again the same as that of the placex table. Every place needs an address, a set of surrounding places that describe the location of the place. The set of address places is made up of OSM places themselves. The **place_addressline** table cross-references for each place all the places that make up its address. Two columns define the address relation: * `place_id` - reference to the place being addressed * `address_place_id` - reference to the place serving as an address part The most of the columns cache information from the placex entry of the address part. The exceptions are: * `fromarea` - is true if the address part has an area geometry and can therefore be considered preceise * `isaddress` - is true if the address part should show up in the address output. Sometimes there are multiple places competing for for same address type (e.g. multiple cities) and this field resolves the tie. The **search_name** table contains the search index proper. It saves for each place the terms with which the place can be found. The terms are split into the name itself and all terms that make up the address. The table mirrors some of the columns from placex for faster lookup. Search terms are not saved as strings. Each term is assigned an integer and those integers are saved in the name and address vectors of the search_name table. The **word** table serves as the lookup table from string to such a word ID. The exact content of the word table depends on the [tokenizer](Tokenizers.md) used. ## Address computation tables Next to the main search tables, there is a set of secondary helper tables used to compute the address relations between places. These tables are partitioned. Each country is assigned a partition number in the country_name table (see below) and the data is then split between a set of tables, one for each partition. Note that Nominatim still manually manages partitioned tables instead of using PostgreSQL's native partitioning. The **search_name_X** tables are used to look up streets that appear in the `addr:street` tag. The **location_area_large_X** tables are used to look up larger areas (administrative boundaries and place nodes) either through their geographic closeness or through `addr:*` entries. The **location_road_X** tables are used to find the closest street for a dependent place. All three table cache specific information from the placex table for their selected subset of places: * `keywords` and `name_vector` contain lists of term ids (from the word table) that the full name of the place should match against * `isguess` is true for places that are not described by an area All other columns reflect their counterpart in the placex table. ## Static data tables Nominatim also creates a number of static tables at import: * `nominatim_properties` saves settings that must not be changed after import * `address_levels` save the rank information from the [ranking configuration](../customize/Ranking.md) * `country_name` contains a fallback of names for all countries, their default languages and saves the assignment of countries to partitions. * `country_osm_grid` provides a fallback for country geometries ## Auxiliary data tables Finally there are some table for auxiliary data: * `location_property_tiger` - saves housenumber from the Tiger import. Its layout is similar to that of `location_propoerty_osmline`. * `place_class_*` tables are helper tables to facilitate lookup of POIs by their class and type. They exist because it is not possible to create combined indexes with geometries. --- ## File: docs/develop/Development-Environment.md # Setting up Nominatim for Development This chapter gives an overview how to set up Nominatim for development and how to run tests. !!! Important This guide assumes you develop under the latest version of Debian/Ubuntu. You can of course also use your favourite distribution. You just might have to adapt the commands below slightly, in particular the commands for installing additional software. ## Installing Nominatim The first step is to install Nominatim itself. Please follow the installation instructions in the [Admin section](../admin/Installation.md). You don't need to set up a webserver for development, the webserver that can be started via `nominatim serve` is sufficient. If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup. Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to install an NFS daemon to enable directory sharing between host and guest. The following packages should get you started: sudo apt install vagrant vagrant-libvirt libvirt-daemon nfs-kernel-server ## Prerequisites for testing and documentation The Nominatim test suite consists of behavioural tests (using pytest-bdd) and unit tests (using pytest). It has the following additional requirements: * [flake8](https://flake8.pycqa.org/en/stable/) (CI always runs the latest version from pip) * [mypy](http://mypy-lang.org/) (plus typing information for external libs) * [Python Typing Extensions](https://github.com/python/typing_extensions) (for Python < 3.9) * [pytest](https://pytest.org) * [pytest-asyncio](https://pytest-asyncio.readthedocs.io) * [pytest-bdd](https://pytest-bdd.readthedocs.io) For testing the Python search frontend, you need to install extra dependencies depending on your choice of webserver framework: * [httpx](https://www.python-httpx.org/) (Starlette only) * [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan) (Starlette only) The documentation is built with mkdocs: * [mkdocs](https://www.mkdocs.org/) >= 1.1.2 * [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25 * [mkdocs-material](https://squidfunk.github.io/mkdocs-material/) * [mkdocs-gen-files](https://oprypin.github.io/mkdocs-gen-files/) ### Installing prerequisites on Ubuntu/Debian The Python tools should always be run with the most recent version. The easiest way, to handle these Python dependencies is to run your development from within a virtual environment. ```sh sudo apt install build-essential libsqlite3-mod-spatialite osm2pgsql \ postgresql-postgis postgresql-postgis-scripts \ pkg-config libicu-dev virtualenv ``` #### Using pip Create a virtual environment and install all dependencies: ```sh virtualenv ~/nominatim-dev-venv . ~/nominatim-dev-venv/bin/activate pip install --group runtime --group dev ``` To install dependencies individually: ```sh pip install \ psutil 'psycopg[binary]' PyICU 'SQLAlchemy[asyncio]' \ python-dotenv jinja2 pyYAML \ mkdocs 'mkdocstrings[python]' mkdocs-gen-files mkdocs-material \ pytest pytest-asyncio pytest-bdd flake8 \ types-jinja2 types-markupsafe types-psutil types-psycopg2 \ types-pygments types-pyyaml types-requests types-ujson \ types-urllib3 typing-extensions gunicorn falcon starlette \ uvicorn mypy osmium aiosqlite mwparserfromhell ``` #### Using uv If you prefer, you can use [uv](https://docs.astral.sh/uv/) for a faster development setup. Install uv following the [official instructions](https://docs.astral.sh/uv/getting-started/installation/), then from the Nominatim source directory: ```sh uv sync ``` This creates a virtual environment and installs all Python dependencies automatically. To run commands in the environment: ```sh uv run nominatim --version uv run pytest test/python uv run make lint ``` ### Running Nominatim during development The source code for Nominatim can be found in the `src` directory and can be run in-place. The source directory features a special script `nominatim-cli.py` which does the same as the installed 'nominatim' binary but executes against the code in the source tree. For example: ``` me@machine:~$ cd Nominatim me@machine:~Nominatim$ ./nominatim-cli.py --version Nominatim version 5.1.0-0 ``` Make sure you have activated the virtual environment holding all necessary dependencies. ## Executing Tests All tests are located in the `/test` directory. To run all tests, run make from the source root: ```sh make tests ``` There are also make targets for executing only parts of the test suite. For example to run linting only use: ```sh make lint ``` The possible testing targets are: mypy, lint, pytest, bdd. For more information about the structure of the tests and how to change and extend the test suite, see the [Testing chapter](Testing.md). ## Documentation Pages The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/) To build the documentation run ``` make doc ``` For local testing, you can start webserver: ``` build> make serve-doc [server:296] Serving on http://127.0.0.1:8000 [handlers:62] Start watching changes ``` If you develop inside a Vagrant virtual machine, use a port that is forwarded to your host: ``` build> mkdocs serve --dev-addr 0.0.0.0:8088 [server:296] Serving on http://0.0.0.0:8088 [handlers:62] Start watching changes ``` --- ## File: docs/develop/ICU-Tokenizer-Modules.md # Writing custom token analysis modules for the ICU tokenizer The [ICU tokenizer](../customize/Tokenizers.md#icu-tokenizer) provides a highly customizable method to pre-process and normalize the name information of the input data before it is added to the search index. It comes with a selection of token analyzers which you can use to adapt your installation to your needs. If the provided modules are not enough, you can also provide your own implementations. This section describes the API for token analysis. !!! warning This API is currently in early alpha status. While this API is meant to be a public API on which other token analyzers may be implemented, it is not guaranteed to be stable at the moment. ## Using custom token analysis modules Token analysis names as set in the `analyzer` property may refer to externally supplied modules. There are two ways to include external modules: through a library or from the project directory. To include a module from a library, use the absolute import path as name and make sure the library can be found in your PYTHONPATH. To use a custom module without creating a library, you can put the module somewhere in your project directory and then use the relative path to the file. Include the whole name of the file including the `.py` ending. ## Custom token analysis module ::: nominatim_db.tokenizer.token_analysis.base.AnalysisModule options: heading_level: 6 ::: nominatim_db.tokenizer.token_analysis.base.Analyzer options: heading_level: 6 ### Example: Creating acronym variants for long names The following example of a token analysis module creates acronyms from very long names and adds them as a variant: ``` python class AcronymMaker: """ This class is the actual analyzer. """ def __init__(self, norm, trans): self.norm = norm self.trans = trans def get_canonical_id(self, name): # In simple cases, the normalized name can be used as a canonical id. return self.norm.transliterate(name.name).strip() def compute_variants(self, name): # The transliterated form of the name always makes up a variant. variants = [self.trans.transliterate(name)] # Only create acronyms from very long words. if len(name) > 20: # Take the first letter from each word to form the acronym. acronym = ''.join(w[0] for w in name.split()) # If that leds to an acronym with at least three letters, # add the resulting acronym as a variant. if len(acronym) > 2: # Never forget to transliterate the variants before returning them. variants.append(self.trans.transliterate(acronym)) return variants # The following two functions are the module interface. def configure(rules, normalizer, transliterator): # There is no configuration to parse and no data to set up. # Just return an empty configuration. return None def create(normalizer, transliterator, config): # Return a new instance of our token analysis class above. return AcronymMaker(normalizer, transliterator) ``` Given the name `Trans-Siberian Railway`, the code above would return the full name `Trans-Siberian Railway` and the acronym `TSR` as variant, so that searching would work for both. ## Sanitizers vs. Token analysis - what to use for variants? It is not always clear when to implement variations in the sanitizer and when to write a token analysis module. Just take the acronym example above: it would also have been possible to write a sanitizer which adds the acronym as an additional name to the name list. The result would have been similar. So which should be used when? The most important thing to keep in mind is that variants created by the token analysis are only saved in the word lookup table. They do not need extra space in the search index. If there are many spelling variations, this can mean quite a significant amount of space is saved. When creating additional names with a sanitizer, these names are completely independent. In particular, they can be fed into different token analysis modules. This gives a much greater flexibility but at the price that the additional names increase the size of the search index. --- ## File: docs/develop/Indexing.md # Indexing Places In Nominatim, the word __indexing__ refers to the process that takes the raw OpenStreetMap data from the place table, enriches it with address information and creates the search indexes. This section explains the basic data flow. ## Initial import After osm2pgsql has loaded the raw OSM data into the place table, the data is copied to the final search tables placex and location_property_osmline. While they are copied, some basic properties are added: * country_code, geometry_sector and partition * initial search and address rank In addition the column `indexed_status` is set to `1` marking the place as one that needs to be indexed. All this happens in the triggers `placex_insert` and `osmline_insert`. ## Indexing The main work horse of the data import is the indexing step, where Nominatim takes every place from the placex and location_property_osmline tables where the indexed_status != 0 and computes the search terms and the address parts of the place. The indexing happens in three major steps: 1. **Data preparation** - The indexer gets the data for the place to be indexed from the database. 2. **Search name processing** - The prepared data is given to the tokenizer which computes the search terms from the names and potentially other information. 3. **Address processing** - The indexer then hands the prepared data and the tokenizer information back to the database via an `INSERT` statement which also sets the indexed_status to `0`. This triggers the update triggers `placex_update`/`osmline_update` which do the work of computing address parts and filling all the search tables. When computing the address terms of a place, Nominatim relies on the processed search names of all the address parts. That is why places are processed in rank order, from smallest rank to largest. To ensure correct handling of linked place nodes, administrative boundaries are processed before all other places. Apart from these restrictions, each place can be indexed independently from the others. This allows a large degree of parallelization during the indexing. It also means that the indexing process can be interrupted at any time and will simply pick up where it left of when restarted. ### Data preparation The data preparation step computes and retrieves all data for a place that might be needed for the next step of processing the search name. That includes * location information (country code) * place classification (class, type, ranks) * names (including names of linked places) * address information (`addr:*` tags) Data preparation is implemented in pl/PgSQL mostly in the functions `placex_indexing_prepare()` and `get_interpolation_address()`. #### `addr:*` tag inheritance Nominatim has limited support for inheriting address tags from a building to POIs inside the building. This only works when the address tags are on the building outline. Any rank 30 object inside such a building or on its outline inherits all address tags when it does not have any address tags of its own. The inheritance is computed in the data preparation step. ### Search name processing The prepared place information is handed to the tokenizer next. This is a Python module responsible for processing the names from both name and address terms and building up the word index from them. The process is explained in more detail in the [Tokenizer chapter](Tokenizers.md). ### Address processing Finally, the preprocessed place information and the results of the search name processing are written back to the database. At this point the update trigger of the placex/location_property_osmline tables take over and fill all the dependent tables. This makes up the most work-intensive part of the indexing. Nominatim distinguishes between dependent and independent places. **Dependent places** are all places on rank 30: house numbers, POIs etc. These places don't have a full address of their own. Instead they are attached to a parent street or place and use the information of the parent for searching and displaying information. Everything else are **independent places**: streets, parks, water bodies, suburbs, cities, states etc. They receive a full address on their own. The address processing for both types of places is very different. #### Independent places To compute the address of an independent place Nominatim searches for all places that cover the place to compute the address for at least partially. For places with an area, that area is used to check for coverage. For place nodes an artificial square area is computed according to the rank of the place. The lower the rank the lager the area. The `location_area_large_X` tables are there to facilitate the lookup. All places that can function as the address of another place are saved in those tables. `addr:*` and `isin:*` tags are taken into account to compute the address, too. Nominatim will give preference to places with the same name as in these tags when looking for places in the vicinity. If there are no matching place names at all, then the tags are at least added to the search index. That means that the names will not be shown in the result as the 'address' of the place, but searching by them still works. Independent places are always added to the global search index `search_name`. #### Dependent places Dependent places skip the full address computation for performance reasons. Instead they just find a parent place to attach themselves to. By default a POI or house number will be attached to the closest street. That can be any major or minor street indexed by Nominatim. In the default configuration that means that it can attach itself to a footway but only when it has a name. When the dependent place has an `addr:street` tag, then Nominatim will first try to find a street with the same name before falling back to the closest street. There are also addresses in OSM, where the housenumber does not belong to a street at all. These have an `addr:place` tag. For these places, Nominatim tries to find a place with the given name in the indexed places with an address rank between 16 and 25. If none is found, then the dependent place is attached to the closest place in that category and the addr:place name is added as *unlisted* place, which indicates to Nominatim that it needs to add it to the address output, no matter what. This special case is necessary to cover addresses that don't really refer to an existing object. When an address has both the `addr:street` and `addr:place` tag, then Nominatim assumes that the `addr:place` tag in fact should be the city part of the address and give the POI the usual street number address. Dependent places are only added to the global search index `search_name` when they have either a name themselves or when they have address tags that are not covered by the places that make up their address. The latter ensures that addresses are always searchable by those address tags. --- ## File: docs/develop/overview.md # Basic Architecture Nominatim provides geocoding based on OpenStreetMap data. It uses a PostgreSQL database as a backend for storing the data. There are three basic parts to Nominatim's architecture: the data import, the address computation and the search frontend. The __data import__ stage reads the raw OSM data and extracts all information that is useful for geocoding. This part is done by osm2pgsql, the same tool that can also be used to import a rendering database. It uses the special flex output style defined in the directory `/lib-lua`. The result of the import can be found in the database table `place`. The __address computation__ or __indexing__ stage takes the data from `place` and adds additional information needed for geocoding. It ranks the places by importance, links objects that belong together and computes addresses and the search index. Most of this work is done in PL/pgSQL via database triggers and can be found in the files in the `sql/functions/` directory. The __search frontend__ implements the actual API. It takes search and reverse geocoding queries from the user, looks up the data and returns the results in the requested format. This part is located in the `nominatim-api` package. The source code can be found in `src/nominatim_api`. --- ## File: docs/develop/Tokenizers.md # Tokenizers The tokenizer is the component of Nominatim that is responsible for analysing names of OSM objects and queries. Nominatim provides different tokenizers that use different strategies for normalisation. This page describes how tokenizers are expected to work and the public API that needs to be implemented when creating a new tokenizer. For information on how to configure a specific tokenizer for a database see the [tokenizer chapter in the Customization Guide](../customize/Tokenizers.md). ## Generic Architecture ### About Search Tokens Search in Nominatim is organised around search tokens. Such a token represents string that can be part of the search query. Tokens are used so that the search index does not need to be organised around strings. Instead the database saves for each place which tokens match this place's name, address, house number etc. To be able to distinguish between these different types of information stored with the place, a search token also always has a certain type: name, house number, postcode etc. During search an incoming query is transformed into a ordered list of such search tokens (or rather many lists, see below) and this list is then converted into a database query to find the right place. It is the core task of the tokenizer to create, manage and assign the search tokens. The tokenizer is involved in two distinct operations: * __at import time__: scanning names of OSM objects, normalizing them and building up the list of search tokens. * __at query time__: scanning the query and returning the appropriate search tokens. ### Importing The indexer is responsible to enrich an OSM object (or place) with all data required for geocoding. It is split into two parts: the controller collects the places that require updating, enriches the place information as required and hands the place to Postgresql. The collector is part of the Nominatim library written in Python. Within Postgresql, the `placex_update` trigger is responsible to fill out all secondary tables with extra geocoding information. This part is written in PL/pgSQL. The tokenizer is involved in both parts. When the indexer prepares a place, it hands it over to the tokenizer to inspect the names and create all the search tokens applicable for the place. This usually involves updating the tokenizer's internal token lists and creating a list of all token IDs for the specific place. This list is later needed in the PL/pgSQL part where the indexer needs to add the token IDs to the appropriate search tables. To be able to communicate the list between the Python part and the pl/pgSQL trigger, the `placex` table contains a special JSONB column `token_info` which is there for the exclusive use of the tokenizer. The Python part of the tokenizer returns a structured information about the tokens of a place to the indexer which converts it to JSON and inserts it into the `token_info` column. The content of the column is then handed to the PL/pqSQL callbacks of the tokenizer which extracts the required information. Usually the tokenizer then removes all information from the `token_info` structure, so that no information is ever persistently saved in the table. All information that went in should have been processed after all and put into secondary tables. This is however not a hard requirement. If the tokenizer needs to store additional information about a place permanently, it may do so in the `token_info` column. It just may never execute searches over it and consequently not create any special indexes on it. ### Querying At query time, Nominatim builds up multiple _interpretations_ of the search query. Each of these interpretations is tried against the database in order of the likelihood with which they match to the search query. The first interpretation that yields results wins. The interpretations are encapsulated in the `SearchDescription` class. An instance of this class is created by applying a sequence of _search tokens_ to an initially empty SearchDescription. It is the responsibility of the tokenizer to parse the search query and derive all possible sequences of search tokens. To that end the tokenizer needs to parse the search query and look up matching words in its own data structures. ## Tokenizer API The following section describes the functions that need to be implemented for a custom tokenizer implementation. !!! warning This API is currently in early alpha status. While this API is meant to be a public API on which other tokenizers may be implemented, the API is far away from being stable at the moment. ### Directory Structure Nominatim expects two files containing the Python part of the implementation: * `src/nominatim_db/tokenizer/_tokenizer.py` contains the tokenizer code used during import and * `src/nominatim_api/search/_tokenizer.py` has the code used during query time. `` is a unique name for the tokenizer consisting of only lower-case letters, digits and underscore. A tokenizer also needs to install some SQL functions. By convention, these should be placed in `lib-sql/tokenizer`. If the tokenizer has a default configuration file, this should be saved in `settings/_tokenizer.`. ### Configuration and Persistence Tokenizers may define custom settings for their configuration. All settings must be prefixed with `NOMINATIM_TOKENIZER_`. Settings may be transient or persistent. Transient settings are loaded from the configuration file when Nominatim is started and may thus be changed at any time. Persistent settings are tied to a database installation and must only be read during installation time. If they are needed for the runtime then they must be saved into the `nominatim_properties` table and later loaded from there. ### The Python modules #### `src/nominatim_db/tokenizer/` The import Python module is expected to export a single factory function: ```python def create(dsn: str, data_dir: Path) -> AbstractTokenizer ``` The `dsn` parameter contains the DSN of the Nominatim database. The `data_dir` is a directory in the project directory that the tokenizer may use to save database-specific data. The function must return the instance of the tokenizer class as defined below. #### `src/nominatim_api/search/` The query-time Python module must also export a factory function: ``` python def create_query_analyzer(conn: SearchConnection) -> AbstractQueryAnalyzer ``` The `conn` parameter contains the current search connection. See the [library documentation](../library/Low-Level-DB-Access.md#searchconnection-class) for details on the class. The function must return the instance of the tokenizer class as defined below. ### Python Tokenizer Class All tokenizers must inherit from `nominatim_db.tokenizer.base.AbstractTokenizer` and implement the abstract functions defined there. ::: nominatim_db.tokenizer.base.AbstractTokenizer options: heading_level: 6 ### Python Analyzer Class ::: nominatim_db.tokenizer.base.AbstractAnalyzer options: heading_level: 6 ### Python Query Analyzer Class ::: nominatim_api.search.query_analyzer_factory.AbstractQueryAnalyzer options: heading_level: 6 ### PL/pgSQL Functions The tokenizer must provide access functions for the `token_info` column to the indexer which extracts the necessary information for the global search tables. If the tokenizer needs additional SQL functions for private use, then these functions must be prefixed with `token_` in order to ensure that there are no naming conflicts with the SQL indexer code. The following functions are expected: ```sql FUNCTION token_get_name_search_tokens(info JSONB) RETURNS INTEGER[] ``` Return an array of token IDs of search terms that should match the name(s) for the given place. These tokens are used to look up the place by name and, where the place functions as part of an address for another place, by address. Must return NULL when the place has no name. ```sql FUNCTION token_get_name_match_tokens(info JSONB) RETURNS INTEGER[] ``` Return an array of token IDs of full names of the place that should be used to match addresses. The list of match tokens is usually more strict than search tokens as it is used to find a match between two OSM tag values which are expected to contain matching full names. Partial terms should not be used for match tokens. Must return NULL when the place has no name. ```sql FUNCTION token_get_housenumber_search_tokens(info JSONB) RETURNS INTEGER[] ``` Return an array of token IDs of house number tokens that apply to the place. Note that a place may have multiple house numbers, for example when apartments each have their own number. Must be NULL when the place has no house numbers. ```sql FUNCTION token_normalized_housenumber(info JSONB) RETURNS TEXT ``` Return the house number(s) in the normalized form that can be matched against a house number token text. If a place has multiple house numbers they must be listed with a semicolon as delimiter. Must be NULL when the place has no house numbers. ```sql FUNCTION token_is_street_address(info JSONB) RETURNS BOOLEAN ``` Return true if this is an object that should be parented against a street. Only relevant for objects with address rank 30. ```sql FUNCTION token_has_addr_street(info JSONB) RETURNS BOOLEAN ``` Return true if there are street names to match against for finding the parent of the object. ```sql FUNCTION token_has_addr_place(info JSONB) RETURNS BOOLEAN ``` Return true if there are place names to match against for finding the parent of the object. ```sql FUNCTION token_matches_street(info JSONB, street_tokens INTEGER[]) RETURNS BOOLEAN ``` Check if the given tokens (previously saved from `token_get_name_match_tokens()`) match against the `addr:street` tag name. Must return either NULL or FALSE when the place has no `addr:street` tag. ```sql FUNCTION token_matches_place(info JSONB, place_tokens INTEGER[]) RETURNS BOOLEAN ``` Check if the given tokens (previously saved from `token_get_name_match_tokens()`) match against the `addr:place` tag name. Must return either NULL or FALSE when the place has no `addr:place` tag. ```sql FUNCTION token_addr_place_search_tokens(info JSONB) RETURNS INTEGER[] ``` Return the search token IDs extracted from the `addr:place` tag. These tokens are used for searches by address when no matching place can be found in the database. Must be NULL when the place has no `addr:place` tag. ```sql FUNCTION token_get_address_keys(info JSONB) RETURNS SETOF TEXT ``` Return the set of keys for which address information is provided. This should correspond to the list of (relevant) `addr:*` tags with the `addr:` prefix removed or the keys used in the `address` dictionary of the place info. ```sql FUNCTION token_get_address_search_tokens(info JSONB, key TEXT) RETURNS INTEGER[] ``` Return the array of search tokens for the given address part. `key` can be expected to be one of those returned with `token_get_address_keys()`. The search tokens are added to the address search vector of the place, when no corresponding OSM object could be found for the given address part from which to copy the name information. ```sql FUNCTION token_matches_address(info JSONB, key TEXT, tokens INTEGER[]) ``` Check if the given tokens match against the address part `key`. __Warning:__ the tokens that are handed in are the lists previously saved from `token_get_name_search_tokens()`, _not_ from the match token list. This is an historical oddity which will be fixed at some point in the future. Currently, tokenizers are encouraged to make sure that matching works against both the search token list and the match token list. ```sql FUNCTION token_get_postcode(info JSONB) RETURNS TEXT ``` Return the postcode for the object, if any exists. The postcode must be in the form that should also be presented to the end-user. ```sql FUNCTION token_strip_info(info JSONB) RETURNS JSONB ``` Return the part of the `token_info` field that should be stored in the database permanently. The indexer calls this function when all processing is done and replaces the content of the `token_info` column with the returned value before the trigger stores the information in the database. May return NULL if no information should be stored permanently. --- ## File: docs/customize/Country-Settings.md # Customizing Per-Country Data Whenever an OSM is imported into Nominatim, the object is first assigned a country. Nominatim can use this information to adapt various aspects of the address computation to the local customs of the country. This section explains how country assignment works and the principal per-country localizations. ## Country assignment Countries are assigned on the basis of country data from the OpenStreetMap input data itself. Countries are expected to be tagged according to the [administrative boundary schema](https://wiki.openstreetmap.org/wiki/Tag:boundary%3Dadministrative): a OSM relation with `boundary=administrative` and `admin_level=2`. Nominatim uses the country code to distinguish the countries. If there is no country data available for a point, then Nominatim uses the fallback data imported from `data/country_osm_grid.sql.gz`. This was computed from OSM data as well but is guaranteed to cover all countries. Some OSM objects may also be located outside any country, for example a buoy in the middle of the ocean. These object do not get any country assigned and get a default treatment when it comes to localized handling of data. ## Per-country settings ### Global country settings The main place to configure settings per country is the file `settings/country_settings.yaml`. This file has one section per country that is recognised by Nominatim. Each section is tagged with the country code (in lower case) and contains the different localization information. Only countries which are listed in this file are taken into account for computations. For example, the section for Andorra looks like this: ``` partition: 35 languages: ca names: !include country-names/ad.yaml postcode: pattern: "(ddd)" output: AD\1 ``` The individual settings are described below. #### `partition` Nominatim internally splits the data into multiple tables to improve performance. The partition number tells Nominatim into which table to put the country. This is purely internal management and has no effect on the output data. The default is to have one partition per country. #### `languages` A comma-separated list of ISO-639 language codes of default languages in the country. These are the languages used in name tags without a language suffix. Note that this is not necessarily the same as the list of official languages in the country. There may be officially recognised languages in a country which are only ever used in name tags with the appropriate language suffixes. Conversely, a non-official language may appear a lot in the name tags, for example when used as an unofficial Lingua Franca. List the languages in order of frequency of appearance with the most frequently used language first. It is not recommended to add languages when there are only very few occurrences. If only one language is listed, then Nominatim will 'auto-complete' the language of names without an explicit language-suffix. #### `names` List of names of the country and its translations. These names are used as a baseline. It is always possible to search countries by the given names, no matter what other names are in the OSM data. They are also used as a fallback when a needed translation is not available. !!! Note The list of names per country is currently fairly large because Nominatim supports translations in many languages per default. That is why the name lists have been separated out into extra files. You can find the name lists in the file `settings/country-names/.yaml`. The names section in the main country settings file only refers to these files via the special `!include` directive. #### `postcode` Describes the format of the postcode that is in use in the country. When a country has no official postcodes, set this to no. Example: ``` ae: postcode: no ``` When a country has a postcode, you need to state the postcode pattern and the default output format. Example: ``` bm: postcode: pattern: "(ll)[ -]?(dd)" output: \1 \2 ``` The **pattern** is a regular expression that describes the possible formats accepted as a postcode. The pattern follows the standard syntax for [regular expressions in Python](https://docs.python.org/3/library/re.html#regular-expression-syntax) with two extra shortcuts: `d` is a shortcut for a single digit([0-9]) and `l` for a single ASCII letter ([A-Z]). Use match groups to indicate groups in the postcode that may optionally be separated with a space or a hyphen. For example, the postcode for Bermuda above always consists of two letters and two digits. They may optionally be separated by a space or hyphen. That means that Nominatim will consider `AB56`, `AB 56` and `AB-56` spelling variants for one and the same postcode. Never add the country code in front of the postcode pattern. Nominatim will automatically accept variants with a country code prefix for all postcodes. The **output** field is an optional field that describes what the canonical spelling of the postcode should be. The format is the [regular expression expand syntax](https://docs.python.org/3/library/re.html#re.Match.expand) referring back to the bracket groups in the pattern. Most simple postcodes only have one spelling variant. In that case, the **output** can be omitted. The postcode will simply be used as is. In the Bermuda example above, the canonical spelling would be to have a space between letters and digits. !!! Warning When your postcode pattern covers multiple variants of the postcode, then you must explicitly state the canonical output or Nominatim will not handle the variations correctly. ### Other country-specific configuration There are some other configuration files where you can set localized settings according to the assigned country. These are: * [Place ranking configuration](Ranking.md) Please see the linked documentation sections for more information. --- ## File: docs/customize/Import-Styles.md # Configuring the Import of OSM data In the very first step of a Nominatim import, OSM data is loaded into the database. Nominatim uses [osm2pgsql](https://osm2pgsql.org) for this task. It comes with a [flex style](https://osm2pgsql.org/doc/manual.html#the-flex-output) specifically tailored to filter and convert OSM data into Nominatim's internal data representation. Nominatim ships with a few preset configurations for this import, each results in a geocoding database of different detail. The [Import section](../admin/Import.md#filtering-imported-data) explains these default configurations in detail. If you want to have more control over which OSM data is added to the database, you can also create your own custom style. Create a new lua style file, put it into your project directory and then set `NOMINATIM_IMPORT_STYLE` to the name of the file. Custom style files can be used to modify the existing preset configurations or to implement your own configuration from scratch. The remainder of the page describes how the flex style works and how to customize it. ## The `flex-base` lua module The core of Nominatim's flex import configuration is the `flex-base` module. It defines the table layout used by Nominatim and provides standard implementations for the import callbacks that help with customizing how OSM tags are used by Nominatim. Every custom style must include this module to make sure that the correct tables are created. Thus start your custom style as follows: ``` lua local flex = require('flex-base') ``` ### Using preset configurations If you want to start with one of the existing presets, then you can import its settings using the `load_topic()` function: ``` lua local flex = require('flex-base') flex.load_topic('streets') ``` The `load_topic` function takes an optional second configuration parameter. The available options are explained in the [themepark section](#using-osm2pgsql-themepark). Available topics are: `admin`, `street`, `address`, `full`. These topic correspond to the [import styles](../admin/Import.md#filtering-imported-data) you can choose during import. To start with the 'extratags' style, use the `full` topic with the appropriate config parameter: ``` lua flex.load_topic('full', {with_extratags = true}) ``` !!! note You can also directly import the preset style files, e.g. `local flex = require('import-street')`. It is not possible to set extra configuration this way. ### How processing works When Nominatim processes an OSM object, it looks for four kinds of tags: The _main tags_ classify what kind of place the OSM object represents. One OSM object can have more than one main tag. In such case one database entry is created for each main tag. _Name tags_ represent searchable names of the place. _Address tags_ are used to compute the address information of the place. Address tags are used for searching and for creating a display name of the place. _Extra tags_ are any tags that are not directly related to search but contain interesting additional information. These are just saved in the database and may be returned with the result [on request](../api/Search.md#output-details). !!! danger Some tags in the extratags category are used by Nominatim to better classify the place. These tags will always be added, independent of any settings in the style. Configuring the style means deciding which key and/or key/value is used in which category. ## Changing the recognized tags The flex style offers a number of functions to set the classification of each OSM tag. Most of these functions can also take a preset string instead of a tag description. These presets describe common configurations that are also used in the definition of the predefined styles. This section lists the configuration functions and the accepted presets. #### Key match lists Some of the following functions take _key match lists_. These lists can contain three kinds of strings to match against tag keys: A string that ends in an asterisk `*` is a prefix match and accordingly matches against any key that starts with the given string (minus the `*`). A suffix match can be defined similarly with a string that starts with a `*`. Any other string is matched exactly against tag keys. ### Main tags `set/modify_main_tags()` allow to define which tags are used as main tags. It takes a lua table parameter which defines for keys and key/value combinations, how they are classified. The following classifications are recognized: | classification | meaning | | :-------------- | :------ | | always | Unconditionally use this tag as a main tag. | | named | Consider as main tag, when the object has a primary name (see [names](#name-tags) below) | | named_with_key | Consider as main tag, when the object has a primary name with a domain prefix. For example, if the main tag is `bridge=yes`, then it will only be added as an extra entry, if there is a tag `bridge:name[:XXX]` for the same object. If this property is set, all names that are not domain-specific are ignored. | | fallback | Consider as main tag only when no other main tag was found. Fallback always implies `named`, i.e. fallbacks are only tried for objects with primary names. | | postcode_area | Tag indicates a postcode area. Copy area into the table of postcodes but only when the object is a relation and has a postcode tagged. | | delete | Completely ignore the tag in any further processing | | extra | Move the tag to extratags and then ignore it for further processing | | ``| Advanced handling, see [below](#advanced-main-tag-handling) | Each key in the table parameter defines an OSM tag key. The value may be directly a classification as described above. Then the tag will be considered a main tag for any possible value that is not further defined. To further restrict which values are acceptable, give a table with the permitted values and their kind of main tag. If the table contains a simple value without key, then this is used as default for values that are not listed. `set_main_tags()` will completely replace the current main tag configuration with the new configuration. `modify_main_tags()` will merge the new configuration with the existing one. Merging is done at value level. For example, when the current setting is `highway = {'always', primary = 'named'}`, then `set_main_tags{highway = 'delete'}` will result in a rule `highway = {'delete', primary = 'named'}`. !!! example ``` lua local flex = require('import-full') flex.set_main_tags{ boundary = {administrative = 'named'}, highway = {'always', street_lamp = 'named', no = 'delete'}, landuse = 'fallback' } ``` In this example an object with a `boundary` tag will only be included when it has a value of `administrative`. Objects with `highway` tags are always included with two exceptions: the troll tag `highway=no` is deleted on the spot. And when the value is `street_lamp` then the object must also have a name, to be included. Finally, if a `landuse` tag is present then it will be used independently of the concrete value when neither boundary nor highway tags were found and the object is named. ##### Presets | Name | Description | | :----- | :---------- | | admin | Basic tag set collecting places and administrative boundaries. This set is needed also to ensure proper address computation and should therefore always be present. You can disable selected place types like `place=locality` after adding this set, if they are not relevant for your use case. | | all_boundaries | Extends the set of recognized boundaries and places to all available ones. | | natural | Tags for natural features like rivers and mountain peaks. | | street/default | Tags for streets. Major streets are always included, minor ones only when they have a name. | | street/car | Tags for all streets that can be used by a motor vehicle. | | street/all | Includes all highway features named and unnamed. | | poi/delete | Adds most POI features with and without name. Some frequent but very domain-specific values are excluded by deleting them. | | poi/extra | Like 'poi/delete' but excluded values are moved to extratags. | ##### Advanced main tag handling The groups described above are in fact only a preset for a filtering function that is used to make the final decision how a pre-selected main tag is entered into Nominatim's internal table. To further customize handling you may also supply your own filtering function. The function takes up to three parameters: a Place object of the object being processed, the key of the main tag and the value of the main tag. The function may return one of three values: * `nil` or `false` causes the entry to be ignored * the Place object causes the place to be added as is * `Place.copy(names=..., address=..., extratags=...) causes the place to be enter into the database but with name/address/extratags set to the given different values. The Place object has some read-only values that can be used to determine the handling: * **object** is the original OSM object data handed in by osm2pgsql * **admin_level** is the content of the admin_level tag, parsed into an integer and normalized to a value between 0 and 15 * **has_name** is a boolean indicating if the object has a primary name tag * **names** is a table with the collected list of name tags * **address** is a table with the collected list of address tags * **extratags** is a table with the collected list of additional tags to save !!! example ``` lua local flex = require('flex-base') flex.add_topic('street') local function no_sidewalks(place, k, v) if place.object.tags.footway == 'sidewalk' then return false end -- default behaviour is to have all footways return place end flex.modify_main_tags(highway = {'footway' = no_sidewalks} ``` This script adds a custom handler for `highway=footway`. It only includes them in the database, when the object doesn't have a tag `footway=sidewalk` indicating that it is just part of a larger street which should already be indexed. Note that it is not necessary to check the key and value of the main tag because the function is only used for the specific main tag. ### Ignored tags The function `ignore_keys()` sets the `delete` classification for keys. This function takes a _key match list_ so that it is possible to exclude groups of keys. Note that full matches always take precedence over suffix matches, which in turn take precedence over prefix matches. !!! example ``` lua local flex = require('flex-base') flex.add_topic('admin') flex.ignore_keys{'old_name', 'old_name:*'} ``` This example uses the `admin` preset with the exception that names that are no longer are in current use, are ignored. ##### Presets | Name | Description | | :----- | :---------- | | metatags | Tags with meta information about the OSM tag like source, notes and import sources. | | name | Non-names that actually describe properties or name parts. These names can throw off search and should always be removed. | | address | Extra `addr:*` tags that are not useful for Nominatim. | ### Tags for `extratags` The function `add_for_extratags()` sets the `extra` classification for keys. This function takes a _key match list_ so that it is possible to move groups of keys to extratags. Note that full matches always take precedence over suffix matches, which in turn take precedence over prefix matches. !!! example ``` lua local flex = require('flex-base') flex.add_topic('street') flex.add_for_extratags{'surface', 'access', 'vehicle', 'maxspeed'} ``` This example uses the `street` preset but adds a couple of tags that are of interest about the condition of the street. ##### Presets Accepts all [presets from ignored tags](#presets_1). ### General pre-filtering _(deprecated)_ `set_prefilters()` allows to set the `delete` and `extra` classification for main tags. This function removes all previously set main tags with `delete` and `extra` classification and then adds the newly defined tags. `set_prefilters()` takes a table with four optional fields: * __delete_keys__ is a _key match list_ for tags that should be deleted * __delete_tags__ contains a table of tag keys pointing to a list of tag values. Tags with matching key/value pairs are deleted. * __extra_keys__ is a _key match list_ for tags which should be saved into extratags * __extra_tags__ contains a table of tag keys pointing to a list of tag values. Tags with matching key/value pairs are moved to extratags. !!! danger "Deprecation warning" Use of this function should be replaced with `modify_main_tags()` to set the data from `delete_tags` and `extra_tags`, with `ignore_keys()` for the `delete_keys` parameter and with `add_for_extratags()` for the `extra_keys` parameter. ### Name tags `set/modify_name_tags()` allow to define the tags used for naming places. Name tags can only be selected by their keys. The import script distinguishes between primary and auxiliary names. A primary name is the given name of a place. Having a primary name makes a place _named_. This is important for main tags that are only included when a name is present. Auxiliary names are identifiers like references. They may be searched for but should not be included on their own. The functions take a table with two optional fields `main` and `extra`. They take _key match lists_ for primary and auxiliary names respectively. A third field `house` can contain tags for names that appear in place of house numbers in addresses. This field can only contain complete key names. 'house tags' are special in that they cause the OSM object to be added to the database independently of the presence of other main tags. `set_name_tags()` overwrites the current configuration, while `modify_name_tags()` replaces the fields that are given. (Be aware that the fields are replaced as a whole. `main = {'foo_name'}` will cause `foo_name` to become the only recognized primary name. Any previously defined primary names are forgotten.) !!! example ``` lua local flex = require('flex-base') flex.set_main_tags{highway = {traffic_light = 'named'}} flex.set_name_tags{main = {'name', 'name:*'}, extra = {'ref'} } ``` This example creates a search index over traffic lights but will only include those that have a common name and not those which just have some reference ID from the city. ##### Presets | Name | Description | | :----- | :---------- | | core | Basic set of recognized names for all places. | | address | Additional names useful when indexing full addresses. | | poi | Extended set of recognized names for pois. Use on top of the core set. | ### Address tags `set/modify_address_tags()` defines the tags that will be used to build up the address of an object. Address tags can only be chosen by their key. The functions take a table with arbitrary fields, each defining a key list or _key match list_. Some fields have a special meaning: | Field | Type | Description | | :---------| :-------- | :-----------| | main | key list | Tags that make a full address object out of the OSM object. This is usually the house number or variants thereof. If a main address tag appears, then the object will always be included, if necessary with a fallback of `place=house`. If the key has a prefix of `addr:` or `is_in:` this will be stripped. | | extra | key match list | Supplementary tags for addresses, tags like `addr:street`, `addr:city` etc. If the key has a prefix of `addr:` or `is_in:` this will be stripped. | | interpolation | key list | Tags that identify address interpolation lines. | | country | key match list | Tags that may contain the country the place is in. The first found value with a two-letter code will be accepted, all other values are discarded. | | _other_ | key match list | Summary field. If a key matches the key match list, then its value will be added to the address tags with the name of the field as key. If multiple tags match, then an arbitrary one wins. | `set_address_tags()` overwrites the current configuration, while `modify_address_tags()` replaces the fields that are given. (Be aware that the fields are replaced as a whole.) !!! example ``` lua local flex = require('import-full') flex.set_address_tags{ main = {'addr:housenumber'}, extra = {'addr:*'}, postcode = {'postal_code', 'postcode', 'addr:postcode'}, country = {'country_code', 'ISO3166-1'} } ``` In this example all tags which begin with `addr:` will be saved in the address tag list. If one of the tags is `addr:housenumber`, the object will fall back to be entered as a `place=house` in the database unless there is another interested main tag to be found. Tags with keys `country_code` and `ISO3166-1` are saved with their value under `country` in the address tag list. The same thing happens to postcodes, they will always be saved under the key `postcode` thus normalizing the multitude of keys that are used in the OSM database. ##### Presets | Name | Description | | :----- | :---------- | | core | Basic set of tags needed to recognize address relationship for any place. Always include this. | | houses | Additional set of tags needed to recognize proper addresses | ### Handling of unclassified tags `set_unused_handling()` defines what to do with tags that remain after all tags have been classified using the functions above. There are two ways in which the function can be used: `set_unused_handling(delete_keys = ..., delete_tags = ...)` deletes all keys that match the descriptions in the parameters and moves all remaining tags into the extratags list. `set_unused_handling(extra_keys = ..., extra_tags = ...)` moves all tags matching the parameters into the extratags list and then deletes the remaining tags. For the format of the parameters see the description in `set_prefilters()` above. When no special handling is set, then unused tags will be discarded with one exception: place tags are kept in extratags for administrative boundaries. When using a custom setting, you should also make sure that the place tag is added for extratags. !!! example ``` lua local flex = require('import-full') flex.set_address_tags{ main = {'addr:housenumber'}, extra = {'addr:*', 'tiger:county'} } flex.set_unused_handling{delete_keys = {'tiger:*'}} ``` In this example all remaining tags except those beginning with `tiger:` are moved to the extratags list. Note that it is not possible to already delete the tiger tags with `set_prefilters()` because that would remove tiger:county before the address tags are processed. ## Filling additional tables Most of the OSM objects are saved in the main `place` table for further processing. In addition to that, there are some smaller tables that save specialised information. The content of these tables can be customized as well. ### Entrance table The table `place_entrance` saves information about OSM nodes that represent an entrance. This data is later mingled with buildings and other areas and can be returned [on request](../api/Search.md#output-details). The table saves the type of entrance as well as a set of custom extra tags. The function `set_entrance_filter()` can be used to customize the table's content. When called without any parameter, then filling the entrance table will be disabled. When called with a preset name, the appropriate preset will be applied. To create a custom configuration, call the function with a table with the following fields: * __main_tags__ is a list of tags that mark an entrance node. The value of the first tag found in the list will be used as the entrance type. * __extra_include__ is an optional list of tags to be added to the extratags for this entrance. When left out, all tags except for the ones defined in 'main_tags' will be included. To disable saving of extra tags, set this to the empty list. * __extra_exclude__ defines an optional list of tags to drop before including the remaining tags as extratags. Note that the tags defined in 'main_tags' will always be excluded, independently of this setting. To have even more fine-grained control over the output, you can also hand in a callback for processing entrance information. The callback function receives a single parameter, the [osm2pgsql object](https://osm2pgsql.org/doc/manual.html#processing-callbacks). This object itself must not be modified. The callback should return either `nil` when the object is not an entrance. Or it returns a table with a mandatory `entrance` field containing a string with the type of entrance and an optional `extratags` field with a simple key-value table of extra information. ##### Presets | Name | Description | | :----- | :---------- | | default | Standard configuration used with `full` and `extratags` styles. | ## Customizing osm2pgsql callbacks osm2pgsql expects the flex style to implement three callbacks, one process function per OSM type. If you want to implement special handling for certain OSM types, you can override the default implementations provided by the flex-base module. ### Enabling additional relation types OSM relations can represent very diverse [types of real-world objects](https://wiki.openstreetmap.org/wiki/Key:type). To be able to process them correctly, Nominatim needs to understand how to create a geometry for each type. By default, the script knows how to process relations of type `multipolygon`, `boundary` and `waterway`. All other relation types are ignored. To add other types relations, set `RELATION_TYPES` for the type to the kind of geometry that should be created. The following kinds of geometries can be used: * __relation_as_multipolygon__ creates a (Multi)Polygon from the ways in the relation. If the ways do not form a valid area, then the object is silently discarded. * __relation_as_multiline__ creates a (Multi)LineString from the ways in the relations. Ways are combined as much as possible without any regards to their order in the relation. !!! Example ``` lua local flex = require('import-full') flex.RELATION_TYPES['site'] = flex.relation_as_multipolygon ``` With this line relations of `type=site` will be included in the index according to main tags found. This only works when the site relation resolves to a valid area. Nodes in the site relation are not part of the geometry. ### Adding additional logic to processing functions The default processing functions are also exported by the flex-base module as `process_node`, `process_way` and `process_relation`. These can be used to implement your own processing functions with some additional processing logic. !!! Example ``` lua local flex = require('import-full') function osm2pgsql.process_relation(object) if object.tags.boundary ~= 'administrative' or object.tags.admin_level ~= '2' then flex.process_relation(object) end end ``` This example discards all country-level boundaries and uses standard handling for everything else. This can be useful if you want to use your own custom country boundaries. ### Customizing the main processing function !!! danger "Deprecation Warning" The style used to allow overwriting the internal processing function `process_tags()`. While this is currently still possible, it is no longer encouraged and may stop working in future versions. The internal `Place` class should now be considered read-only. ## Using osm2pgsql-themepark The Nominatim osm2pgsql style is designed so that it can also be used as a theme for [osm2pgsql-themepark](https://osm2pgsql.org/themepark/). This makes it easy to combine Nominatim with other projects like [openstreetmap-carto](https://github.com/gravitystorm/openstreetmap-carto) in the same database. To set up one of the preset styles, simply include a topic with the same name: ``` local themepark = require('themepark') themepark:add_topic('nominatim/address') ``` Themepark topics offer two configuration options: * **street_theme** allows to choose one of the sub topics for streets: * _default_ - include all major streets and named minor paths * _car_ - include all streets physically usable by cars * _all_ - include all major streets and minor paths * **with_extratags**, when set to a truthy value, then tags that are not specifically used for address or naming are added to the extratags column The customization functions described in the [Changing recognized tags](#changing-the-recognized-tags) section are available from the theme. To access the theme you need to explicitly initialize it. !!! Example ``` lua local themepark = require('themepark') themepark:add_topic('nominatim/full', {with_extratags = true}) local flex = themepark:init_theme('nominatim') flex.modify_main_tags{'amenity' = { 'waste_basket' = 'delete'} } ``` This example uses the full Nominatim configuration but disables importing waste baskets. You may also write a new configuration from scratch. Simply omit including a Nominatim topic and only call the required customization functions. Customizing the osm2pgsql processing functions as explained [above](#adding-additional-logic-to-processing-functions) is not possible when running under themepark. Instead include other topics that make the necessary modifications or add an additional processor before including the Nominatim topic. !!! Example ``` lua local themepark = require('themepark') local function discard_country_boundaries(object) if object.tags.boundary == 'administrative' and object.tags.admin_level == '2' then return 'stop' end end themepark:add_proc('relation', discard_country_boundaries) -- Order matters here. The topic needs to be added after the custom callback. themepark:add_topic('nominatim/full', {with_extratags = true}) ``` Discarding country-level boundaries when running under themepark. ## Changing the style of existing databases There is usually no issue changing the style of a database that is already imported and now kept up-to-date with change files. Just be aware that any change in the style applies to updates only. If you want to change the data that is already in the database, then a reimport is necessary.