Messages posted by: ccrosby

Question from a user on the EarthScope lidar point cloud data:

Im using the Lidar Cloud Point Set from the EarthScope Intermountain Seismic Belt LiDAR Project.

The problem is that the LAS files or ASCII files have no information on the "point return" class. Only come with "Intensity" besides XYZ information.

Was this bug fixed in other data sets but not in the EarthScope Intermountain Seismic Belt LiDAR Project?

Could you update me on this issue?

Hi Jason,
Thanks for your interest in OpenTopography.

The simple answer is that we currently use both relational database and flat file systems in OpenTopography.

In the early days of the project, OpenTopography received all point cloud data in ascii format. We explored various relational database configurations on different types of hardware to optimize hosting of those data (single and multi-partition; spatial and non-spatial indexing; supercomputer and commodity hardware). Currently, the production system that hosts those older ascii datasets is an 8-node partitioned IBM DB2 database. Full details on our database exploration work can be found in:

Nandigam, V., Baru, C., and Crosby, C.J., 2010, Database Design for High-Resolution LIDAR Topography Data, in, M. Gertz and B. Ludascher (Eds.): SSDBM 2010, Lecture Notes in Computer Science 6187, pp. 151159.

Now that most lidar point cloud data we receive comes in LAS format, we've built a hybrid database and flat file system to manage these data. Metadata about the LAS files we receive are stored in the DB. A query against the DB returns the appropriate files which are then clipped and merged as necessary using the open source libLAS libraries.

There are significant advantages and disadvantages to both systems. If you have more specific questions feel free to send me an email (ccrosby@sdsc.edu) or give a call (858.822.5458 ), and I can answer questions or put you in contact with the correct team members here.

We also have an active project ("Performance Evaluation of On-Demand Provisioning of Data Intensive Applications" aka CloudStor) which is exploring cloud-based lidar data management and comparing cloud cost and performance to database solutions. A summary of the CloudStor project is on the OpenTopography Cyberinfrastructure research page.

Thanks,

-Chris Crosby

Reply:

Good question. This is a common problem that people working with terrestrial laser scanners (TLS) data encounter. The issue isnt as common with airborne lidar since most applications of the data are either performed with the point cloud or 2.5D surfaces. For your data youll probably need to look at software specifically designed for handling TLS data. ESRI isnt going to get you there I dont think. Packages like Polyworks (http://www.innovmetric.com/polyworks/3D-scanners/home.aspx?lang=en), RiScan Pro ( http://www.riegl.com/index.php?id=221 - companion to RIEGLs scanners), and Leicas Cyclone ( http://hds.leica-geosystems.com/en/Leica-Cyclone_6515.htm) are designed for constructing models from 3D scanner data. You may also find that there are options in the CAD space (e.g. AutoCAD 3D Studio Max, Geomagic, etc.) since theyre using TLS more and more for urban modeling and other engineering-type applications and historically are familiar with 3D modeling. If your colleague owns the scanner from which these data came, he may very well already have one of the TLS software packages above since they typically accompany the scanner when you purchase it.

For volumetric data representation, netCDF is a good format, and were been using it for seismic tomography and other truly 3d geophysical data. I havent explored how youd get TLS lidar into netCDF though.

An OT user recently asked the following question via email. Question and response posted here for the posterity of others with similar challenges.

-cc

I'm working with a colleague who has LIDAR point clouds of terrain and
wants to model them as true 3D for volumetric calculations. The problem
is that there are surface cavities (sea caves or cliff overhangs, for
instance) that aren't handled properly by a 2.5D representation...I'm open to any suggestions for this - ESRI or otherwise. Any suggestions would be really helpful...

Hello,
Kepler can be used to orchestrate services and tools, so it is quite possible to build a workflow using Kepler that calls various LiDAR data processing tools. The predecessor to OpenTopography, the GEON LiDAR Workflow (GLW), used Kepler to control and chain together the processes that made it possible to query, extract, grid, and visualize LiDAR point cloud data online. For more information on how Kepler was used in the GLW see: http://www.springerlink.com/content/240m180l141rq744/

Good luck,

-cc

chent wrote:

First, When I run the Fusion with the mode of ""color by pulse number" , the left column displays the attribute with "Height". According to your answer, it should be the point collected time in GPS time system. I don't know the relationship of them.

I'm sorry, but I am not a regular user of Fusion, so specific questions about how to use the software are not possible for me to answer. There is good documentation on the software available from the Fusion website: http://www.fs.fed.us/eng/rsac/fusion/

chent wrote:

Second, in your answer, you tell me this parameter is significant for "identifying the geometry of the data acquisition, amount of swath overlap, and places where you may have edge artifacts from swath edges". I don't know the exact meaning of it, because I am not very familiar with LIDAR. So would you please give me some advices on selecting some reference papers about this questions(If possible, you can mail them to my e-mail chent@cea.gov.cn). And I can have more details of LIDAR data without always bothering you.

To learn more about LiDAR, I recommend you look at the short course materials that we make available here: http://www.opentopography.org/index.php/resources/short_courses
These courses (specifically the two courses held in 2009) included an intro to LiDAR technology and other basics. If you work through the slides and tutorials you should develop a better understanding.

Good luck,

-cc

"Pulse number" is another way of saying collection time. Typically, each lidar point has an attribute that records when the point was collected (in GPS time). So, in FUSION, sorting by pulse number effectively sorts pulses based on GPS time and displays data from adjacent swaths by color. Sorting the data by collection time can be useful for identifying the geometry of the data acquisition, amount of swath overlap, and places where you may have edge artifacts from swath edges.

Hope this is helpful.

-Chris

FYI - I just encountered this problem again with a 4GB zip file we received. IZArc is the only thing that will open it.

-cc

Ajay,
Sorry about the delayed reply to this question - Ive been traveling the past week+ and am a bit backlogged.

A few comments that may be helpful:
1. The 1.4 m shot spacing is likely an average and will vary significantly throughout the data set. Are you working with only bare earth points by any chance? If so, the spacing of those returns is likely far greater than 1.4 m if the vegetation in the area is significant (the 1.4 m spec. likely applies to all points, not the bare earth ones).

2. There is no precise rule about grid resolution as a function of return spacing, but ideally youd like each grid cell to be representative of at least a single elevation value. In most cases with lidar data, the cell value is calculated from a few points. P2G essentially assumes that grid res. is > shot spacing (i.e. 1 m DEM from 3 shot/m data), and thus it is appropriate to perform a local operation on the points (e.g. Take the mean values in a search area around the grid cell center). If your shot spacing is > grid resolution, youll need to use a true interpolator like a spline or kriging to fit a surface to the points and to fill the holes. Take a look at these pages to help you wrap your head around how P2G works:
http://lidar.asu.edu/KnowledgeBase/GLW_Search_Radius/
http://lidar.asu.edu/KnowledgeBase/WCptcount/

3. I recommend you run your data through the point count option in P2G. This will generate a grid of shots per search area and help you to decide what is the optimum grid resolution and search radius in P2G to generate a grid from your point data. Note that if P2G does not find a point within the search radius, it populates the grid with a NULL/NoData value. We have recently updated the P2G algorithm to fill these nulls, but we havent released the code yet. Hopefully it will be online in the next month or two.

Hope this is helpful let me know if you have more questions.

-cc

Ajay provided a bit more info on the problem via email:

I'm working with a dataset from Texas with a nominal point spacing of 1.4 meters. I converted the .las files to ASCII and produced a grid using the desktop version of GEON Points2Grid. I made the DEM with a grid resolution of 1.4 meters. The output raster that contains many "NoData" pixels (see attachment for screenshot - the spacing between pixel groups is on the scale of a meter). I double-checked the input file, and it looks like it's in the right format (northing,easting,elevation). So I'm wondering (1) is there a rule of thumb that relates point density to grid resolution, and (2) do you know what might be going awry with the grid generation?

Answer:

A ".TGZ file" is a compressed archive file format similar to the more familiar .ZIP. OpenTopography writes out custom DEM data and point cloud files as .tar.gz files to make the files smaller and therefore easier to download. Files downloaded from OpenTopography will look something like this:
1266365984583406150039.tar.gz
Where the long number string is a unique identifier for the job, and the .tar.gz indicates that it is a tar file compressed with gzip compression

To work with a custom DEM product produced in OpenTopography, you first need to uncompress the .tar.gz file to access the .arc.asc (the actual DEM) file which you will import via ArcToolbox. I prefer the free WinRAR software to uncompress the .tar.gz files that OT generates, but there are many other utilities out there than can work with these files.

-cc

Here is a question I received via email - it is a simple problem but one that many other users may also encounter:

I am trying to import the custom DEM into ArcMap, and the site says that I can just do:

Conversion Tools > To Raster > ASCII to Raster ; with the output data type set to float.

However, this conversion requires a file type of .txt or .arc, and the file downloaded from the custom DEM email is a 'TGZ' file with the extension ".tar". How do I convert this to a form which can be imported into ArcMap through the ASCII to Raster conversion tool?

The Letters from the SAL blog (Spatial Analysis Laboratory at UVM) has a nice ~10 minute video video tutorial on using Quick Terrain Modeler to convert LAS point cloud data to a raster, perform some simple raster editing, and then export the DEM to a GeoTiff for use in ArcGIS for further analysis.

Hi Linda,
As I said during the course last week, software for do it yourself point cloud classification ("filtering" if you prefer) is relatively limited. There are the big, expensive, commercial software packages like Terrasolid (http://www.terrasolid.fi/) and MARS (http://merrick.com/index.php/services/geospatial-solutions/mars-software) that are designed for production environments (i.e. for use by a lidar vendor).

Another commercial alternative is Tiffs (http://globalidar.com) which I know virtually nothing about.

In the free / open source realm, GRASS GIS claims to have robust point cloud classification capability. Take a look at: http://grass.osgeo.org/wiki/LIDAR
I've used GRASS in the past, but haven't explored the lidar classification routine yet. It is on my long list of things to do.

Finally, you might take a look at this paper:
Evans, J.S., and A.T. Hudak. (2007) A multiscale curvature algorithm for classifying discrete return lidar in forested environments. IEEE Transactions on Geoscience and Remote Sensing 45(4):1029-1038.
Download at: http://www.treesearch.fs.fed.us/pubs/29032

I hear that they are working on a C++ version of the code, but it is currently not available.

Hope this is helpful, keep us posted on your progress exploring the various options.

Cheers,

-cc

Winzip should be able to handle it, but I generally use WinRAR (http://www.rarlab.com/) since it is free and can handle a variety of compressed formats. The native windows compression utility will not work on the tar.gz files. Let me know if winrar doesn't get the job done.

-cc