geospacial-engineering/search/search_index.json

{"config":{"lang":["en"],"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Article Series: Part 1: Setup PostGIS using Docker Part 2: Connecting QGIS to PostGIS and importing data Part 3: Querying Data Part 4: Advanced Analytics Terms and Tools: Term Description GIS Geographic Information System (manage, analyze and present geographic data) Enterprise GIS GIS system that spans the entire organization Geospacial Database Examples include PostgreSQL PostGIS, Oracle Spacial, SQLite's Spatialite or MSSQL Spacial Desktop GIS Examples include ESRI ArcGIS and QGIS QGIS Desktop GIS Software PostgreSQL Open Source RDBMS Server PostGIS Geospecial Extension to PostgreSQL PgAdmin4 PostgreSQL Web based SQL Editor and Administration Tool Docker We'll use the Docker container platform to run the software. Docker Commpose Compose multiple applications. Docker Hub Publish the image to docker hub. Links and Reference: Github Repository with Dockerfile and docker-compose.yaml Docker Image: Reference and further documentation: http://workshops.boundlessgeo.com/postgis-intro/ https://postgis.net/docs/manual-2.4/ https://boundlessgeo.com/gis-solutions/ Fancy math with Hilbert curves: http://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/","title":"Geospacial Engineering with Docker, PostgreGIS and QGIS"},{"location":"#article-series","text":"Part 1: Setup PostGIS using Docker Part 2: Connecting QGIS to PostGIS and importing data Part 3: Querying Data Part 4: Advanced Analytics","title":"Article Series:"},{"location":"#terms-and-tools","text":"Term Description GIS Geographic Information System (manage, analyze and present geographic data) Enterprise GIS GIS system that spans the entire organization Geospacial Database Examples include PostgreSQL PostGIS, Oracle Spacial, SQLite's Spatialite or MSSQL Spacial Desktop GIS Examples include ESRI ArcGIS and QGIS QGIS Desktop GIS Software PostgreSQL Open Source RDBMS Server PostGIS Geospecial Extension to PostgreSQL PgAdmin4 PostgreSQL Web based SQL Editor and Administration Tool Docker We'll use the Docker container platform to run the software. Docker Commpose Compose multiple applications. Docker Hub Publish the image to docker hub.","title":"Terms and Tools:"},{"location":"#links-and-reference","text":"Github Repository with Dockerfile and docker-compose.yaml Docker Image:","title":"Links and Reference:"},{"location":"#reference-and-further-documentation","text":"http://workshops.boundlessgeo.com/postgis-intro/ https://postgis.net/docs/manual-2.4/ https://boundlessgeo.com/gis-solutions/","title":"Reference and further documentation:"},{"location":"#fancy-math-with-hilbert-curves","text":"http://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/","title":"Fancy math with Hilbert curves:"},{"location":"about/","text":"About Me Mihai Criveti https://ie.linkedin.com/in/crivetimihai","title":"About Me"},{"location":"about/#about-me","text":"","title":"About Me"},{"location":"about/#mihai-criveti","text":"https://ie.linkedin.com/in/crivetimihai","title":"Mihai Criveti"},{"location":"opendata/","text":"A look at Open Data Common GIS data formats: QGIS can be used to import a variety of formats in PostGIS, including Shapefiles, KML and GeoJSON. Shapefile: a popular geospatial vector data format for geographic information system (GIS) software. It is developed and regulated by Esri as a (mostly) open specification for data interoperability among Esri and other GIS software products. KML: Keyhole Markup Language is an XML notation for expressing geographic annotation and visualization within Internet-based, two-dimensional maps and three-dimensional Earth browsers. GeoJSON: a format for encoding a variety of geographic data structures. Open Data Sources Ireland's Open Data Portal: https://data.gov.ie/ Irish Spatial Data Exchange: http://isde.ie Census 2016 Open Data: http://census2016.geohive.ie/ European Data Portal: https://www.europeandataportal.eu 2600+ Open Data sources: https://www.opendatasoft.com/a-comprehensive-list-of-all-open-data-portals-around-the-world/ Interesting datasets: Landslide / Geohazards data, from GSI Railway stations Railway lines Railway road intersections Built up areas Settlements Population Groundwater vulnerability Dublin development plans Downloading data: # Download the data: wget http://spatial.dcenr.gov.ie/GSI_DOWNLOAD/Landslide_Susceptibility_Map_Ireland.zip # Unzip the data: unzip Landslide_Susceptibility_Map_Ireland.zip","title":"Open Data Sources and Types"},{"location":"opendata/#a-look-at-open-data","text":"","title":"A look at Open Data"},{"location":"opendata/#common-gis-data-formats","text":"QGIS can be used to import a variety of formats in PostGIS, including Shapefiles, KML and GeoJSON. Shapefile: a popular geospatial vector data format for geographic information system (GIS) software. It is developed and regulated by Esri as a (mostly) open specification for data interoperability among Esri and other GIS software products. KML: Keyhole Markup Language is an XML notation for expressing geographic annotation and visualization within Internet-based, two-dimensional maps and three-dimensional Earth browsers. GeoJSON: a format for encoding a variety of geographic data structures.","title":"Common GIS data formats:"},{"location":"opendata/#open-data-sources","text":"Ireland's Open Data Portal: https://data.gov.ie/ Irish Spatial Data Exchange: http://isde.ie Census 2016 Open Data: http://census2016.geohive.ie/ European Data Portal: https://www.europeandataportal.eu 2600+ Open Data sources: https://www.opendatasoft.com/a-comprehensive-list-of-all-open-data-portals-around-the-world/","title":"Open Data Sources"},{"location":"opendata/#interesting-datasets","text":"Landslide / Geohazards data, from GSI Railway stations Railway lines Railway road intersections Built up areas Settlements Population Groundwater vulnerability Dublin development plans","title":"Interesting datasets:"},{"location":"opendata/#downloading-data","text":"# Download the data: wget http://spatial.dcenr.gov.ie/GSI_DOWNLOAD/Landslide_Susceptibility_Map_Ireland.zip # Unzip the data: unzip Landslide_Susceptibility_Map_Ireland.zip","title":"Downloading data:"},{"location":"psql/","text":"Using the psql commandline client and other tools List the networks. There should be a container running on the network postgis_default docker network ls 2e2afc387fbb postgis_default bridge local Connect to PostGIS using the psql client (from Docker): docker run --net postgis_default -it --rm \\ --link postgis:postgis postgres psql -h postgis -U postgres -d gis Using a local client: psql -h localhost -U postgres -d gis List Tables: \\dt Run SQL Queries: -- Events created by ABC, in descending order of creation date: SELECT county, quaternary, slope_type, bedrock_ty, land_use_c, creationda, creator FROM public.\"Landslide_Events\" WHERE creator = 'ABC' ORDER BY creationda DESC;","title":"Commandline and psql"},{"location":"psql/#using-the-psql-commandline-client-and-other-tools","text":"","title":"Using the psql commandline client and other tools"},{"location":"psql/#list-the-networks-there-should-be-a-container-running-on-the-network-postgis_default","text":"docker network ls 2e2afc387fbb postgis_default bridge local","title":"List the networks. There should be a container running on the network postgis_default"},{"location":"psql/#connect-to-postgis-using-the-psql-client-from-docker","text":"docker run --net postgis_default -it --rm \\ --link postgis:postgis postgres psql -h postgis -U postgres -d gis","title":"Connect to PostGIS using the psql client (from Docker):"},{"location":"psql/#using-a-local-client","text":"psql -h localhost -U postgres -d gis","title":"Using a local client:"},{"location":"psql/#list-tables","text":"\\dt","title":"List Tables:"},{"location":"psql/#run-sql-queries","text":"-- Events created by ABC, in descending order of creation date: SELECT county, quaternary, slope_type, bedrock_ty, land_use_c, creationda, creator FROM public.\"Landslide_Events\" WHERE creator = 'ABC' ORDER BY creationda DESC;","title":"Run SQL Queries:"},{"location":"qgis/","text":"Loading data with QGIS In this article, I will show you how to: Install QGIS on Ubuntu Linux. Connect QGIS to PostgreSQL/PostGIS. Import data (shapefiles, GeoJSON) into the GIS database using DBManager. 1. Install QGIS: On Ubuntu Linux, you can use: sudo apt-get update sudo apt-get install qgis For other operating systems, follow the instructions listed at https://qgis.org/en/site/ Connect to PostGIS Add PostGIS in QGis Under Browser, Right click PostGIS > New Connection* and select Name: postgis, Host: localhost, Port: 5432. Save the connection details. 2. Import Landslide shapefile data into QGIS, then Posgres Import the data in QGIS First, we import the sample data into QGIS: 'Layer > Data Source Manager > Home' and find the layers Select the layers you wish to add and click the Add Selected Layers button. Export the data to PostgreSQL / PostGIS Click on Database > DB Manager > DB Manager Select *PostGIS - yourdb > your schema > Table > Import Layer / File and name it (ex: Landslide_Events) Repeat step 2 for every layer you wish to import Close the DB Manager Note: The import activity can take a long time. You can monitor progress in the PGAdmin4 Dashboard, by looking at the Tuples In: Inserts graph: Delete the layers - and load them from the DB In Layers, right click each layer - Remove In Browser > Postgis > postgis > public - double click each layer (in the right order).","title":"Load data with QGIS"},{"location":"qgis/#loading-data-with-qgis","text":"In this article, I will show you how to: Install QGIS on Ubuntu Linux. Connect QGIS to PostgreSQL/PostGIS. Import data (shapefiles, GeoJSON) into the GIS database using DBManager.","title":"Loading data with QGIS"},{"location":"qgis/#1-install-qgis","text":"On Ubuntu Linux, you can use: sudo apt-get update sudo apt-get install qgis For other operating systems, follow the instructions listed at https://qgis.org/en/site/","title":"1. Install QGIS:"},{"location":"qgis/#connect-to-postgis","text":"Add PostGIS in QGis Under Browser, Right click PostGIS > New Connection* and select Name: postgis, Host: localhost, Port: 5432. Save the connection details.","title":"Connect to PostGIS"},{"location":"qgis/#2-import-landslide-shapefile-data-into-qgis-then-posgres","text":"","title":"2. Import Landslide shapefile data into QGIS, then Posgres"},{"location":"qgis/#import-the-data-in-qgis","text":"First, we import the sample data into QGIS: 'Layer > Data Source Manager > Home' and find the layers Select the layers you wish to add and click the Add Selected Layers button.","title":"Import the data in QGIS"},{"location":"qgis/#export-the-data-to-postgresql-postgis","text":"Click on Database > DB Manager > DB Manager Select *PostGIS - yourdb > your schema > Table > Import Layer / File and name it (ex: Landslide_Events) Repeat step 2 for every layer you wish to import Close the DB Manager Note: The import activity can take a long time. You can monitor progress in the PGAdmin4 Dashboard, by looking at the Tuples In: Inserts graph:","title":"Export the data to PostgreSQL / PostGIS"},{"location":"qgis/#delete-the-layers-and-load-them-from-the-db","text":"In Layers, right click each layer - Remove In Browser > Postgis > postgis > public - double click each layer (in the right order).","title":"Delete the layers - and load them from the DB"},{"location":"setup/","text":"Docker PostGIS and PGAdmin In this article, I will show you how to: Create a Postgis docker image FROM postgres and publish it to hub.docker.com . Create a Geospacial Database environment in Docker Compose with PostGIS and PGAdmin4. Use PGAdmin4 to create a database an enable Geospacial extensions. 1. Build a PostGIS Docker Image A Dockerfile contains instructions to build an image. Start by extending an existing PostgreSQL 10 Debian image: FROM postgres:10 . Use apt-get to install required PostGIS extensions. Use docker build to create the image, then push it to docker hub. Creating the Dockerfile # Extend exiting PostreSQL 10 Debian image: https://hub.docker.com/_/postgres/ FROM postgres:10 MAINTAINER Mihai Criveti # Install PostGIS packages RUN apt-get update RUN apt-get install --no-install-recommends --yes \\ postgresql-10-postgis-2.4 postgresql-10-postgis-2.4-scripts postgresql-contrib Building the image Turn the Dockerfile into a usable image using docker build . Tag the image with a namespace (the one used on Docker Hub): cmihai docker build --tag cmihai/postgis . Uploading the image to Docker Hub Push the Dockerfile, README and docker-compose.yaml examples to github Test the image end to end Push the image to docker hub export DOCKER_ID_USER=\"username\" docker login docker push docker tag cmihai/postgis $DOCKER_ID_USER/my_image docker push $DOCKER_ID_USER/my_image 2. Composing multiple images with docker compose PGAdmin4 is a web based PostgreSQL Administration and SQL Development environment. Docker Compose can link an existing dpage/pgadmin4 image from Docker Hub to cmihai/postgis Login to http://localhost:5050 admin : admin after running docker-compose up Create docker-compose.yaml version: '3.1' services: postgis: image: cmihai/postgis container_name: postgis ports: - '5432:5432' environment: POSTGRES_PASSWORD: postgres volumes: - pgdata:/var/lib/postgresql/data pgadmin4: image: dpage/pgadmin4 container_name: pgadmin4 ports: - '5050:80' environment: PGADMIN_DEFAULT_EMAIL: admin PGADMIN_DEFAULT_PASSWORD: admin links: - postgis volumes: pgdata: Starting the services Use docker-compose up to start the services: docker-compose up 3. Create a database and enable PostGIS with PGAdmin4 Login to pgadmin4: http://localhost:5050 with admin:admin Add a connection to postgis with user/pass postgres : postgres Create a new database and call it gis Open the SQL Query Tool on the newly created gis database: In the Browser window, select Servers > postgis > Databases > gis , the run Tools > Query Tool from the Menu . Run the following SQL code to enable postgis database extensions: -- Enable PostGIS (includes raster) CREATE EXTENSION postgis; -- Enable Topology CREATE EXTENSION postgis_topology; -- Enable PostGIS Advanced 3D and other geoprocessing algorithms CREATE EXTENSION postgis_sfcgal; -- Fuzzy matching needed for Tiger CREATE EXTENSION fuzzystrmatch; -- Rule based standardizer CREATE EXTENSION address_standardizer; -- Example rule data set CREATE EXTENSION address_standardizer_data_us; -- Enable US Tiger Geocoder CREATE EXTENSION postgis_tiger_geocoder; Expected Outcome: gis database with geospacial extensions Query returned successfully : gis > Extensions now lists a number of GIS extensions: postgis, postgis_sfgal, postgis_tiger_geocoder, postgis_topology, fuzzystrmatch, address_standardizer and address_standardizer_data_us . gis > Schema > public > Functions has been populated with a high number (1000+) of GIS specific functions. A new table called spacial_ref_sys is now available under gis > Schemas > public > Tables . New schemas: tiger , tiger_data and topology have been created. Next Steps: Load Geospacial data from shapefile, KML, GeoJSON, etc. Connect GIS Desktop clients such as QGIS. Connect to PostGIS using Python (ex: geopandas). Perform geospacial queries and analysis on the data. Links and Reference: Github Repository with Dockerfile and docker-compose.yaml: https://github.com/crivetimihai/geospacial-engineering Docker Image: https://hub.docker/com","title":"Docker PostGIS and PGAdmin4"},{"location":"setup/#docker-postgis-and-pgadmin","text":"In this article, I will show you how to: Create a Postgis docker image FROM postgres and publish it to hub.docker.com . Create a Geospacial Database environment in Docker Compose with PostGIS and PGAdmin4. Use PGAdmin4 to create a database an enable Geospacial extensions.","title":"Docker PostGIS and PGAdmin"},{"location":"setup/#1-build-a-postgis-docker-image","text":"A Dockerfile contains instructions to build an image. Start by extending an existing PostgreSQL 10 Debian image: FROM postgres:10 . Use apt-get to install required PostGIS extensions. Use docker build to create the image, then push it to docker hub.","title":"1. Build a PostGIS Docker Image"},{"location":"setup/#creating-the-dockerfile","text":"# Extend exiting PostreSQL 10 Debian image: https://hub.docker.com/_/postgres/ FROM postgres:10 MAINTAINER Mihai Criveti # Install PostGIS packages RUN apt-get update RUN apt-get install --no-install-recommends --yes \\ postgresql-10-postgis-2.4 postgresql-10-postgis-2.4-scripts postgresql-contrib","title":"Creating the Dockerfile"},{"location":"setup/#building-the-image","text":"Turn the Dockerfile into a usable image using docker build . Tag the image with a namespace (the one used on Docker Hub): cmihai docker build --tag cmihai/postgis .","title":"Building the image"},{"location":"setup/#uploading-the-image-to-docker-hub","text":"Push the Dockerfile, README and docker-compose.yaml examples to github Test the image end to end Push the image to docker hub export DOCKER_ID_USER=\"username\" docker login docker push docker tag cmihai/postgis $DOCKER_ID_USER/my_image docker push $DOCKER_ID_USER/my_image","title":"Uploading the image to Docker Hub"},{"location":"setup/#2-composing-multiple-images-with-docker-compose","text":"PGAdmin4 is a web based PostgreSQL Administration and SQL Development environment. Docker Compose can link an existing dpage/pgadmin4 image from Docker Hub to cmihai/postgis Login to http://localhost:5050 admin : admin after running docker-compose up","title":"2. Composing multiple images with docker compose"},{"location":"setup/#create-docker-composeyaml","text":"version: '3.1' services: postgis: image: cmihai/postgis container_name: postgis ports: - '5432:5432' environment: POSTGRES_PASSWORD: postgres volumes: - pgdata:/var/lib/postgresql/data pgadmin4: image: dpage/pgadmin4 container_name: pgadmin4 ports: - '5050:80' environment: PGADMIN_DEFAULT_EMAIL: admin PGADMIN_DEFAULT_PASSWORD: admin links: - postgis volumes: pgdata:","title":"Create docker-compose.yaml"},{"location":"setup/#starting-the-services","text":"Use docker-compose up to start the services: docker-compose up","title":"Starting the services"},{"location":"setup/#3-create-a-database-and-enable-postgis-with-pgadmin4","text":"Login to pgadmin4: http://localhost:5050 with admin:admin Add a connection to postgis with user/pass postgres : postgres Create a new database and call it gis Open the SQL Query Tool on the newly created gis database: In the Browser window, select Servers > postgis > Databases > gis , the run Tools > Query Tool from the Menu . Run the following SQL code to enable postgis database extensions: -- Enable PostGIS (includes raster) CREATE EXTENSION postgis; -- Enable Topology CREATE EXTENSION postgis_topology; -- Enable PostGIS Advanced 3D and other geoprocessing algorithms CREATE EXTENSION postgis_sfcgal; -- Fuzzy matching needed for Tiger CREATE EXTENSION fuzzystrmatch; -- Rule based standardizer CREATE EXTENSION address_standardizer; -- Example rule data set CREATE EXTENSION address_standardizer_data_us; -- Enable US Tiger Geocoder CREATE EXTENSION postgis_tiger_geocoder;","title":"3. Create a database and enable PostGIS with PGAdmin4"},{"location":"setup/#expected-outcome-gis-database-with-geospacial-extensions","text":"Query returned successfully : gis > Extensions now lists a number of GIS extensions: postgis, postgis_sfgal, postgis_tiger_geocoder, postgis_topology, fuzzystrmatch, address_standardizer and address_standardizer_data_us . gis > Schema > public > Functions has been populated with a high number (1000+) of GIS specific functions. A new table called spacial_ref_sys is now available under gis > Schemas > public > Tables . New schemas: tiger , tiger_data and topology have been created.","title":"Expected Outcome: gis database with geospacial extensions"},{"location":"setup/#next-steps","text":"Load Geospacial data from shapefile, KML, GeoJSON, etc. Connect GIS Desktop clients such as QGIS. Connect to PostGIS using Python (ex: geopandas). Perform geospacial queries and analysis on the data.","title":"Next Steps:"},{"location":"setup/#links-and-reference","text":"Github Repository with Dockerfile and docker-compose.yaml: https://github.com/crivetimihai/geospacial-engineering Docker Image: https://hub.docker/com","title":"Links and Reference:"}]}