I've been attempting to create a TopoJson file with consolidated layer data containing, among other layers, U.S. States, Counties, and Congressional Districts.
Original .shp shapefiles come from the Census Bureau's Cartographic Boundary Files.
These were converted to GeoJson via ogr2ogr.
Then combined into TopoJson format via the node server side library, with quantization of 1e7 and retain-proportion of 0.15. Up to this point there is no indication of any problem.
I view the final topojson file using mapshaper and things seem to look OK:
But, when attempting to render with the topojson client library and D3.geo.path(), I encounter some strange paths in the congressionalDist layer: (notice the large rectangular paths around the continental US, AK, and HI)
A working version of the page can be found here: http://jsl6906.net/D3/topojson_problem/map/
A couple of relevant notes:
If I change my topojson generation script to remove simplification, the paths then seem to show correctly via the same d3.js page
If I only keep the congressionalDist layer when creating the topojson, the paths then seem to show correctly via the same d3.js page:
After attempting as much troubleshooting as I've been able to handle, I figured I would ask someone here to see if someone has experienced any similar issues. Thanks for any help.
As I mentioned in the comments, I had noticed that the three offending rectangles all were bound to data with an id property ending in ZZ, while all other paths had IDs ending with numbers.
After some Google searching, I came up with what I think is the answer.
According to this document on the census.gov website,
In Connecticut, Illinois, and Michigan the state participant did not assign the current (113th)
congressional districts to cover all of the state or equivalent area. The code “ZZ” has been assigned
to areas with no congressional district defined (usually large water bodies). These unassigned areas
are treated within state as a single congressional district for purposes of data presentation.
It seems that these three undefined districts would account for the three rectangles. It is unclear at what point in the process they cause the issue, but I believe there is a simple solution to your immediate problem. While searching for information about the ZZ code, I stumbled across this makefile in a project by mbostock called us-atlas.
It seems he had encountered a similar issue and had managed to filter out the undefined congressional districts when running ogr2ogr. Here is the relevant code from that file:
# remove undefined congressional districts
shp/us/congress-ungrouped.shp: shp/us/congress-unfiltered.shp
rm -f $#
ogr2ogr -f 'ESRI Shapefile' -where "GEOID NOT LIKE '%ZZ'" $# $<
I'm betting that if you run your ogr2ogr on your shapefile using the flags shown here it will solve the problem.
Related
What I'm building
There is this game Path of Exile, that has a passive tree that i'm recreating in my Vue application with help from Pixi Js. The game developers release a json with the data for this passive tree every expansion for people to work with.
The Tree: https://www.pathofexile.com/passive-skill-tree
The json: https://github.com/EmmittJ/SkillTree_TypeScript/blob/master/data/3.12.0-pre/SkillTree.json (Don't forget to press the "View raw" link)
Sandbox i created with a variation where i tried to do it with fetch(): https://codesandbox.io/s/sharp-kalam-hzvt4?fontsize=14&hidenavigation=1&theme=dark
The Problem
While working with this json data and Pixi js to draw this tree i'm running into some performance issues. With all the different variations i tried i either get large memory leaks after a while (on this i'm most concerned), or the whole json is loaded in my production bundle so those files are huge, or both. At this point i tried so much stuff i'm not sure what to do anymore.
What i'm looking for
I'm looking to build a <tree> component in my Vue app with the data from this json that is just static in a /data folder (will change every 3 months at max), and with that data run some functions to draw the passive tree on my canvas. When this tree is drawn i still want to be able to dynamically change things on the canvas. This what the structure of the site should look like for now:
What i tried
There are of course multiple ways to get the json data and work with it, but all seem to run in some kind of issues. I tried using Axios, Fetch, just a import, put it in my state, and put it in a static data object. But in the end i feel like my JS Heap always looks something like this:
Sandbox
I created a sandbox with a variation where i load the data with fetch():
https://codesandbox.io/s/sharp-kalam-hzvt4?fontsize=14&hidenavigation=1&theme=dark
Anyone who can help me out how i should go about this?
I'm ingesting a large geoJSON file in leaflet with somewhere near 19,000 features (points). This results in a massive amount of clutter on the map and is unmanageable.
The goal is to use geolocation to mark my location, draw a 5nm circle around it, and only display geoJSON features that reside within that geometry.
A condensed version of my project is:
https://jsfiddle.net/blintster/d2ucock2/3/
I've worked out finding my location and drawing the circle, but I'm not able to parse the geoJSON features on location. Ideally the output would function like this: https://esri.github.io/esri-leaflet/examples/spatial-queries.html However, that method only seems to apply to L.esri.FeatureLayer and this is a locally imported geoJSON.
The geoJSON layer in question is below where [airports] is the 19,000 entries:
var allairportsLayer = L.geoJson([airports], {
filter: airportFilter,
onEachFeature: function(feature, layer) {
layer.bindPopup(feature.properties.Type + " - " + feature.properties.FacilityName + "<br>Contact Info: " + feature.properties.Manager + "<br> Phone: " + feature.properties.ManagerPhone);
}
}).addTo(map);
function airportFilter(feature) {
if (feature.properties.State === "MD") return true
};
I was able to pair down the results slightly by using the filter method by state but that only allowed me to determine if an attribute meets a specified criteria.
I'v also tried the following method: https://www.mapbox.com/mapbox.js/example/v1.0.0/marker-radius-search/
with no luck.
Does anyone know of any additional methods I could try to parse the data so it only shows points that reside within a geometry?
However, that method only seems to apply to L.esri.FeatureLayer and this is a locally imported geoJSON.
esri-leaflet piggybacks off ArcGIS Server and ArcGIS Online services which provide a backend hosted database that supports spatial queries.
obviously Esri isn't the only option, but your use-case is a perfect example of a situation when it is beneficial not to fetch an entire dataset that you don't plan on displaying.
you could create a arcgis developer account and then sign into arcgis.com to upload your .geojson file as a new hosted service for free.
you could find another hosted service that provides comparable functionality
you could run your own server, install your own PostGIS database and hook up spatial web queries yourself.
you could continue to download all 19,000 features on page load and either:
a) simplify your search and test whether the relevant L.latLngBounds.contains() each point.
b) use something like turf to test the relationship with an actual circle. (one caveat worth mentioning here is that leaflet doesn't include any built in methods for generating actual L.circle geometry so you'd need more custom logic for that too. i wrote something similar here that you are welcome to ripoff).
I downloaded a shape file from the U.S. Census and passed it through http://mapshaper.org/, which gave me a working topojson file to use in D3.
However, I realized the topojson file had all the features labels stripped. So there is no way identify features to link to CSV data.
I installed topojson, but topojson version 2 does not have the same commands as the version one to convert shape files to topojson and retain the features. Previously, topojson -o output.json input.shp This is the current topojson github reference https://github.com/topojson/topojson.
So my question is what is the best way to convert shapefiles to topojson and retain the features attributes.
I'll answer in relation to the mapshaper method you've used.
The problem as I understand it is that you are losing attributes/properties contained within the shapefile when converting to topojson, such as feature id or name which breaks links to data in other files.
When using mapshaper, you need to copy the .dbf, .prj and the .shp files that come with the shapefile into mapshaper. The .dbf contains all the attributes/properties of the features. This will ensure the topojson has the properties that the shapefile does.
If there are too many attributes per feature and you only want one or two of them, then you might need to use another piece of software for that action (or another to both remove attributes/properties and export to topojson).
I have seen all the responses to a similar question, however, they are all either old, or no one has answered them.
I have been given the task to obtain zip codes and display their corresponding boundaries to the user on a Google map, like in this example.
I am writing this code in Javascript and using the Google Maps API. I want the user to input a zip code and a marker drops down on their destination with a border representing that zip code area. I see that Google Maps currently has something in their map code that allows one to see the boundaries if someone puts a zip code on maps.google.com. I have used polygons but this wouldn't help make a border around a certain zip code.
Any suggestions on how to obtain this?
Thanks in advance!
There is not an easy answer to this that I know of. But here is a high level design of how to do it.
All of the shape files for zip codes can be found at the census site and can be downloaded from this ftp server. However, that's a ton of data, so you need a place to store it. I recommend using the PostgreSQL database with the PostGIS add on. It is free and open source and generally awesome. It has a utility for converting .shp files (the type in the census shape files) into PostGIS geometry form. PostGIS let's you retrieve the shapes back out as KML.
You can either a) retrieve a shape from the database as KML when it is needed and display it on the map or b) pre-generate a kml file for every zip code ahead of time and retrieve a file as it is needed (this would take up quite a bit of space).
You need to become familiar with GeoJSON formatted FeatureCollections. You can render them on any set of map tiles with OpenLayers (or probably Google API as well)
This may seem pretty hard, but is totally approachable.
You can purchase GeoJSON files for groups of Zipcodes if you search around.
DOwnload the shapefile from here https://catalog.data.gov/dataset/tiger-line-shapefile-2019-2010-nation-u-s-2010-census-5-digit-zip-code-tabulation-area-zcta5-na
Simplifying using GDAL
We can use the ogr2ogr command from the GDAL library to convert the shapefile to geojson but even with only one field and simple coordinates the output file is over 1GB.
ogr2ogr -f GeoJSON -select ZCTA5CE10 -lco COORDINATE_PRECISION=6 zcta.geojson /vsizip/tl_2017_us_zcta510.zip
I tried to simplify this to topojson, but the topojson library chokes on this even on a very powerful 2017 MacBook Pro.
npx topojson -q 1e4 -o zcta_topo.json zcta.geojson >> JavaScript head out of memory
Another method I tried was using the -simplify option in ogr2ogr. The simplify argument is a unit of measure based on the spatial reference system of the shapefile. Since the srs for the ZCTAs is WGS84 the unit is a lat/lon measure.
ogr2ogr -f "GeoJSON" -lco COORDINATE_PRECISION=6 -select ZCTA5CE10 -simplify 0.006 zcta.geojson /vsizip/tl_2017_us_zcta510.zip
This creates a much smaller GeoJSON file (30MB) which the TopoJSON can easily handle and we end up with a more managable (but still too large) 13MB topojson file. Additionally, the topology of the dataset is very poor at medium to large scales.
npx topojson -q 1e5 -o zcta_topo.json zcta.geojson
Simplifying using Postgis
Create a docker volume to use for persistence
docker volume create postgresql
Run the postgis docker
docker run --name postgis -p 25432:5432 -it --mount source=postgresql,target=/var/lib/postgresql kartoza/postgis
Load the zcta shapefile into postgis
ogr2ogr -f "PostgreSQL" -progress -select "ZCTA5CE10" -overwrite -lco OVERWRITE=yes -nln zcta -nlt PROMOTE_TO_MULTI -t_srs "EPSG:4326" PG:"dbname='gis' host='localhost' port='25432' user='docker' password='docker'" ~/Downloads/tl_2017_us_zcta510/tl_2017_us_zcta510.shp
Sample query with st_simplifypreservetopology (New England). This takes a long time to run for the entire country and we still lose a lot of the topology.
select st_simplifypreservetopology(wkb_geometry, 0.025) as thegeom, zcta5ce10 from zcta where zcta5ce10 like '0%' OR zcta5ce10 like '1%'
Simplifying using Mapshaper (Best solution)
The Mapshaper library can output TopoJSON directly from the shapefile without JavaScript memory heap errors. This command creates a ~6MB topojson file that we can use. It also manages to keep topology very well by assuming that very close verticies and edges should be coincident.
npx -p mapshaper mapshaper-xl tl_2017_us_zcta510.shp snap -simplify 0.1% -filter-fields ZCTA5CE10 -rename-fields zip=ZCTA5CE10 -o format=topojson zcta_mapshaper.json
source:https://github.com/elastic/ems-file-service/issues/6
I want to create a territories map on Google Maps that follows zip code boundaries but have some of my territories include multiple zip codes. I was wondering if there is a way to do this by listing which of google's pre-mapped zip code boundaries (with the hope that Google has a zip code boundary list to pull from?) I would like to combine in each group.
For example:
Territory A:
84000,
84001,
84002,
etc.
Then have,
Territory B:
84108,
84101,
84115,
etc.
And so on covering the entire USA.
Any tips or advice would be great. I'm looking to use pre-defined boundaries so it will be easier to update and any zip code changes country-wide will automatically be accounted for.
Thanks!
Google's Zip Code Boundaries are not available through the API or for download or any other mechanism. That is solely a feature of maps.google.com. To do this, you would have to source the data yourself. There are a number of companies that do this, just search on zip code boundaries download. The US Census provides something called a Zip Code Tabulation Area, which is not the same but related and potentially useful.