LayerMapping data import utility

The LayerMapping class provides a way to map the contents of vector spatial data files (e.g. shapefiles) into GeoDjango models.

This utility grew out of the author’s personal needs to eliminate the code repetition that went into pulling geometries and fields out of a vector layer, converting to another coordinate system (e.g. WGS84), and then inserting into a GeoDjango model.

Note

Use of LayerMapping requires GDAL.

Warning

GIS data sources, like shapefiles, may be very large. If you find that LayerMapping is using too much memory, set DEBUG to False in your settings. When DEBUG is set to True, Django automatically logs every SQL query – and when SQL statements contain geometries, this may consume more memory than is typical.

Example

1. You need a GDAL-supported data source, like a shapefile (here we’re using a simple polygon shapefile, test_poly.shp, with three features):

   >>> from django.contrib.gis.gdal import DataSource
   >>> ds = DataSource('test_poly.shp')
   >>> layer = ds[0]
   >>> print(layer.fields) # Exploring the fields in the layer, we only want the 'str' field.
   ['float', 'int', 'str']
   >>> print(len(layer)) # getting the number of features in the layer (should be 3)
   3
   >>> print(layer.geom_type) # Should be 'Polygon'
   Polygon
   >>> print(layer.srs) # WGS84 in WKT
   GEOGCS["GCS_WGS_1984",
       DATUM["WGS_1984",
           SPHEROID["WGS_1984",6378137,298.257223563]],
       PRIMEM["Greenwich",0],
       UNIT["Degree",0.017453292519943295]]

2. Now we define our corresponding Django model (make sure to use migrate):

   from django.contrib.gis.db import models
   class TestGeo(models.Model):
       name = models.CharField(max_length=25) # corresponds to the 'str' field
       poly = models.PolygonField(srid=4269) # we want our model in a different SRID
       def __str__(self):
           return 'Name: %s' % self.name

3. Use LayerMapping to extract all the features and place them in the database:

   >>> from django.contrib.gis.utils import LayerMapping
   >>> from geoapp.models import TestGeo
   >>> mapping = {'name' : 'str', # The 'name' model field maps to the 'str' layer field.
                  'poly' : 'POLYGON', # For geometry fields use OGC name.
                  } # The mapping is a dictionary
   >>> lm = LayerMapping(TestGeo, 'test_poly.shp', mapping)
   >>> lm.save(verbose=True) # Save the layermap, imports the data.
   Saved: Name: 1
   Saved: Name: 2
   Saved: Name: 3

Here, LayerMapping transformed the three geometries from the shapefile in their original spatial reference system (WGS84) to the spatial reference system of the GeoDjango model (NAD83). If no spatial reference system is defined for the layer, use the source_srs keyword with a SpatialReference object to specify one.

LayerMapping API

class LayerMapping(model, data_source, mapping, layer=0, source_srs=None, encoding=None, transaction_mode=’commit_on_success’, transform=True, unique=True, using=’default’)

The following are the arguments and keywords that may be used during instantiation of LayerMapping objects.

Changed in Django Development version:

Support for pathlib.Path data_source was added.

save() Keyword Arguments

LayerMapping.``save(verbose=False, fid_range=False, step=False, progress=False, silent=False, stream=sys.stdout, strict=False)

The save() method also accepts keywords. These keywords are used for controlling output logging, error handling, and for importing specific feature ranges.

Troubleshooting

Running out of memory

As noted in the warning at the top of this section, Django stores all SQL queries when DEBUG=True. Set DEBUG=False in your settings, and this should stop excessive memory use when running LayerMapping scripts.

MySQL: max_allowed_packet error

If you encounter the following error when using LayerMapping and MySQL:

OperationalError: (1153, "Got a packet bigger than 'max_allowed_packet' bytes")

Then the solution is to increase the value of the max_allowed_packet setting in your MySQL configuration. For example, the default value may be something low like one megabyte – the setting may be modified in MySQL’s configuration file (my.cnf) in the [mysqld] section:

max_allowed_packet = 10M