Pages

Sunday, April 3, 2016

Pure Python NetCDF Reader

Great news! Pure Python geospatial programmer Karim Bahgat created a depenency-free Pythonpyncf.  If you're not familiar with NetCDF, it is technically a collection of software libraries and machine-independent data formats commonly used for multi-dimensional scientific data of any type.  NetCDF currently uses a hierarchial data storage format called HDF5.  NetCDF has a concept of dimensions, variables, and attributes and can scale from one-to-one relationships up to many-to-many relationships in all directions.  In geospatial contexts, NetCDF is frequently used to store 2D and 3D raster or vector datasets over time.  It is very popular for ocean observations and climatological data.
NetCDF reader called

He's posted the code and some more usage information on Github too:  https://github.com/karimbahgat/pyncf

Karim goes into detail on the ideas behind the library on his blog.  But be careful, the examples on the blog aren't as current as the ones on Github: https://thepythongischallenge.wordpress.com/2016/03/26/pynetcdf-netcdf-files-in-pure-python/

As a quick test I installed pyncf using pip directly from Github:
pip install https://github.com/karimbahgat/pyncf/archive/master.zip

I then downloaded the sample time series orthogonal point data NetCDF sample file from here:
https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/

The actual link to the netcdf file is here:
/thredds/fileServer/testdata/netCDFTemplateExamples/timeSeries/BodegaMarineLabBuoy.nc

What's nice about these NOAA samples is that they have a plain-text CDL (Common Data Language) version as well that let's you see what's in the file when you're experimenting with an API like this:
http://data.nodc.noaa.gov/testdata/netCDFTemplateExamples/timeSeries/BodegaMarineLabBuoy.cdl

First, I imported the library and created a NetCDF object from the sample file:

import pyncf
nc = pyncf.NetCDF(filepath="BodegaMarineLabBuoy.nc")
Then I created a variable to hold the header dictionary:
header = nc.header
The header metadata contains a variety of summary data as nested dictionaries and list.  We can access the product description by traversing that structure:

header["gatt_list"][1]["values"]

'These seawater data are collected by a moored fluorescence and turbidity instrument operated at Cordell Bank, California, USA, by CBNMS and BML. Beginning on 2008-04-23, fluorescence and turbidity measurements were collected using a Wetlabs ECO Fluorescence and Turbidity Sensor (ECO-FLNTUSB). The instrument depth of the water quality sensors was 01.0 meter, in an overall water depth of 85 meters (both relative to Mean Sea Level, MSL). The measurements reflect a 10 minute sampling interval.'

The time values in this dataset are stored as epoch seconds.  I accessed those and then converted the first and last into readable dates to see exactly what period this dataset spans:

import time
t = nc.read_dimension_values("time")
print time.ctime(t[0])
'Mon Jul 28 12:30:00 2008'
print time.time(t[-1])
'Wed Sep 10 10:31:00 2008'
The pyncf codebase is considered an alpha version and is currently read only, but what a great addition to your pure Python geospatial toolbox!  I wish this library was available when I updated "Learning Geospatial Analysis with Python"!

Friday, March 11, 2016

New Shapefile Browser

A developer named Adrián Eirís created a lightweight Python shapefile browser whose only compiled dependency is Tkinter which is included with most Python installations or is just a package manager command away.  The tool is called "Ship-Shape-File-Navigator".  It lets you preview a shapefile's geometry and attributes as well as perform common file operations like copying and renaming shapefiles as easily as if you were manipulating a single file (very handy!).  So if you need to do some quick shapefile management and you don't wan to fire up QGIS or ArcMap, check it out.  You can find out more at: http://ShipShapeFileNavigator.org


Here's a quick screenshot showing the geometry preview:

Friday, January 15, 2016

2016 Book Sale!

My publisher, Packt Publishing, started 2016 with a 50% off sale for the eBook versions of both the
first and second editions of "Learning Geospatial Analysis with Python"!  Why would you want both?  The first edition is all Python 2 and the second edition, besides additional code samples and other significant updates, is focused on Python 3.  There are links on the right side of this page or you can just use code LGAPS50 at https://www.packtpub.com.  This deal is good until Jan. 31 and I promise you won't be disappointed.


When the first edition launched in 2013, there was one other geospatial python book out - Erik Westra's excellent "Geospatial Python Development".  Now there are 12 on Packt's site alone and other publishers are working to catch up.  I started this blog because I love Python and I love geospatial analysis but couldn't find much information out there.  Now there is a wealth of excellent geospatial python info and software coming from all directions!

You can help to continue to increase the use of Python in the geospatial field by discussing it online in blogs and forums, linking to interesting software and reading material on Twitter, releasing or contributing software on Github, and buying relevant books.  Even in this digital age, books sales have a tremendous impact on the buzz and therefore the use of a given technology.  And at the end of the day, mind share usually drives an industry to use a given technology.  So however you participate, help spread the word that Python is the best language for geospatial technology (and pretty much everything else!).  Of all the ways of "voting" for Python, buying books is the most point-and-click, dead simple, yet still personally rewarding way to make an impact!

Wednesday, December 30, 2015

First Python 3 Geospatial Book

Chapter 8 demonstrates advanced geospatial modeling 
using Python 3 to create a color hillshade like this as well 
as several other commonly used real-world examples.
The second edition of "Learning Geospatial Analysis with Python" is now available!  To my
knowledge, this is the first book dedicated to Python 3 and geospatial analysis.
The book opens with the incredible story of how the grass-roots, open-source Ushahidi project stopped the Ebola epidemic in its tracks. It continues with one of the best histories of geospatial technologies you'll find.  The chapters closes with you creating a complete GIS using only 60 lines of Python code!

Chapter 2 teaches you the data types you'll encounter on your journey into geospatial analysis. Chapter 3 surveys commonly-used geospatial software to give you the context you need to navigate geospatial solutions.  Chapter 4 covers Python geospatial tools.  Chapter 5 dives into GIS operations with Python.  Chapter 6 teaches you Python 3 remote sensing techniques.  And Chapter 7 is a one-of-a-kind examination of elevation data with several examples.  Chapter 8 covers complete geospatial modeling techniques for real-world situations like the color hillshade in this post.  Chapter 9 explores the emerging world of real-time tracking data.  And Chapter 10 pulls the whole book together to build a complete application.

You get over 100 examples with test data and code plus explanations in either a print or e-book available from Packt Publishing or pretty much any other online book seller.  If you get the print book, or if you just want to see the images, I've made them available as a zip file download here.

While many books in this category do a great job of covering Python APIs to different libraries, this book focuses as much as possible on algorithms to show you the raw calculations that make up geospatial analysis.  This approach allows you to not only run the examples but break things down line by line to see how it works and recombine pieces of code on your own to go beyond the book.

Click the link on the right to check it out on Amazon!