Messages posted by: ccrosby

Thanks for the feedback. Like all things ESRI, I'm sure once you've figured out the necessary steps and worked through the many quirks, it can be quite powerful. When I get some time I'll explore the capability more.

The ESRI Geoprocessing blog has a series of articles (three so far) on how to use ArcGIS to manage LIDAR point cloud data. The articles are quite detailed and are a good resource for folks who want to put their point cloud data into Arc. The articles available so far are:

- Assessing Lidar Coverage and Sample Density
- Creating raster DEMs and DSMs from large lidar point collections
- Data Area Delineation from Lidar Points
- Estimating Forest Density and Height (added 03/23/09)
- Creating Intensity Images from Lidar (added 05/21/09)
- Updating a portion of a terrain dataset with new measurements (added 07/06/09)
- Minimizing noise from lidar for contouring and slope analysis (added 09/02/09)

I've experimented a bit with ArcGIS' LIDAR point cloud capability and generally was frustrated with how long it took to complete the various operations.

I plan to revisit LIDAR point data in Arc using these posts as a guide.

Does anyone use ArcGIS to manage their LIDAR point cloud data? If so, what has been your experience?

Glad to hear that you are finding OpenTopography helpful.

I have a couple of data set questions. For any of the data sets, did the LiDAR sensors have the capability to record both the first and last return pulses and the returned intensity? If yes, is this also available?

All datasets available via OpenTopo are multiple return. You can choose to download (and produce DEMs for) bare earth points, vegetation returns, structure returns, or all returns. We have retained all attributes for the datasets that we host, so if you choose to download the point cloud data, you will find attribution for classification as well as return number. Intensity is an attribute for the B4 dataset as well as all GeoEarthScope data. On the Metadata & Files page of OpenTopo there are links to metadata documents that describe the attributes of the various datasets

Are aerial images or multispectral images available for any of the data sets in addition to the LiDAR data? I know I can google earth the locations and get aerial imagery that way, but I was wondering if on the order of 15cm pixel resolution aerial imagery was available.

The only dataset for which imagery was acquired at the same time as the LIDAR was B4. I've never seen these data, but they do exist. I'm told they were not acquired with a true photogrammetric camera so may be tough to work with. MASTER multispectral imagery has been flown for faults in southern California - more info on those data are in this abstract: http://gsa.confex.com/gsa/2005AM/finalprogram/abstract_95708.htm

A good question via email - reposted here with response for everyone's benefit:

"The website says that the northern CA data are 1 km^2 at 0.5 m resolution (e.g. I pasted in a screen shot of one of the maps). When I download the data however, the bare earth DEMs are 1000x1000 points. It looks like they they do span 1 km^2, so must be spaced at 1 meter, not 0.5..."

You are correct in some places the NoCal data was processed to 1 m DEMs because NCALM determined that there weren't enough ground returns (due to dense canopy and steep terrain) to support 0.5 m data. I've updated the language on the download page to clarify this.

I got this question recently:
"I keep noticing the image at the top of the OpenTopography Portal page - what chunk of lidar is that image from?"

For others who may have the same question, here's the answer:

The image in the banner is from data on the Garlock fault (which, IMO, has spectacular strike-slip geomorph that rivals or exceeds the Carrizo SAF). The attached Google Earth place mark will show you where to find the data. Garlock is only available in 0.5 m DEM tiles, not point cloud yet (the full SoCal GeoES dataset hasn't been delivered yet, just Garlock DEMs). Assuming you've got access to Google Earth, and since you've got the place mark, your best bet to grab the data is to go to http://www.opentopography.org/kml and download the Southern & Eastern California GeoEarthScope LiDAR Hillshades kmz file as well as the Southern & Eastern California GeoEarthScope LiDAR DEM tiles KMZ further down the page. The first file will allow you to browse hillshades of the data and choose what you want, the second file shows the outlines of the 1 km^2 DEM tiles and has direct links to download the DEMs. You can also grab the tiles via the SoCal Google Map interface here: http://opentopography.org/dems

Response to above followup question
----------------
Hey,
So, you've stumbled into another subtle complexity caused by the way the data are organized and delivered by NCALM (the guys who actually did the data acquisition). Tile 574_4129 is in an area of overlap where data was acquired on two separate days (day 088 and day 089). Therefore there are two DEMs for one km^2 tile area. If you click just right on the Google Map or Google Earth tile download interface the button explodes into two buttons, allowing you to download both DEMs available for that tile. In this case, the two urls for tile 574_4129 are:
http://cordillera.la.asu.edu:8080/NoCAL/MapServlet?mapid=574_4129_07_089.ZIP
http://cordillera.la.asu.edu:8080/NoCAL/MapServlet?mapid=574_4129_07_088.ZIP

One of these gives a complete DEM, the other is the edge of an acquisition and hence is incomplete despite appearing to be in the middle of the acquisition. Take a look at the attached screen cap...you'll note that the cyan lines show the edge of the acquisition area and each day of acquisition is a separate polygon. In this case, tile 574_4129 falls inside both the acquisition to the north and the acquisition to the south, hence the two tiles.

More on this issue of incomplete DEM tiles downloaded from OpenTopo - because this is a problem that has been encountered by other users, I am going to continue to post the dialog here.
--------------------------
Christopher,

Thanks for the swift reply. I had been aware of the "edge" issue, but perhaps mistakenly thought it applied only to the edges near the lines indicating the extents of the data collection.

The data missing from 574_4130 is on the "interior" side of the tile, but I realize that still makes it a bit like the case of 502_3938 which you've already addressed in the Forum.

But in the case of 574_4129, it is surrounded by 8 other tiles and is therefore entirely "non-edge" as far as I can tell.

So is it true that the "edge" issue can apply to any tile in the whole collection, regardless of how far it is from the acquisition edges?

My reply (initially via email, reposted here):
-----------------------
Hi,
Those files are not missing data. The geometry of the dataset is generally a long, narrow swath parallel to the major faults in the region. The data are arbitrarily tiled out into 1 km^2 tiles regardless of how much data falls in the tile. So, for tiles that fall on the edge of the data acquisition there may be very little data in a tile. You can see this issue clearly if you overlay the hillshade data on the tile index in Google Earth (both files are available via OpenTopography under "Google Earth files" ( http://www.opentopography.org/kml).

I've discussed this issue in a post on the forum related to the Garlock (GeoES SoCal data) because it is a common misconception among users.
Direct link to the topic related to "data gaps", including a screen capture to explain the issue is here:
http://opentopo.sdsc.edu:8080/jforum-2.1.7-b3/posts/list/27.page

Hope this helps,

-cc

Another question from an OpenTopo user regarding potential data gaps:
--------------
A new Message has been Submitted:

Entry Date: 2008-12-30 12:42 PM
Subject: Report Bug
Message: I'm not sure if this is a bug, or an error on my part ....

Several data files in the OpenTopography (No Cal LiDAR GeoEarthScope) project are apparently missing large chunks of data, even though they are not called out in the metadata or other materials (as far as I can tell) as areas that need to be reflown.

Examples: grid574_4130 and grid574_4129. (And to a lesser extent: grid573_4130, grid574_4131, and grid573_4129.)

Is there really a full 1 km square of 0.5m DEM data in these files, or is this a bug?

Thanks for looking into this.

Max, I think what you are seeing is the edge of the data acquisition. The data were tiled out at 1 sq. km regardless of whether there is data in the whole tile or not. In places along the edge of the acquisition you'll sometimes get tiles with little to no data in them. Check out this screen capture of the tile index overlain by the actual data in the area of tile 502_3938:

As you can see, 502_3938 has just a little bit of data in it, as do many of the other tiles in the area.

Hope this clarifies.

-cc

I get this question regarding the GeoEarthScope "standard" DEMs on a regular basis, so I am going to address it here for everyone:

I have downloaded some of Northern California GeoEarthScope LiDAR DEM tiles in Google Earth. For every single grid, there are two folders with different *.adf files. Can you please clarify what are the files of *.adf format?

First, for each zip file downloaded from either the Google Map or KML (Google Earth) interfaces, there are two DEM files - a bare earth (vegetation removed) and a full feature (vegetation, buildings, etc. still present). The bare earth DEMs are prepended with "fg" (for "filtered grid") and the full feature are preprended with "ug" (for "unfiltered grid"). For example:

When you download file 443_4443.zip and uncompress it, it contains two directories:
fg443_4443
ug443_4443

Inside each of these directories are these files:
dblbnd.adf
hdr.adf
log
prj.adf
sta.adf
w001001.adf
w001001x.adf

Basically, the ESRI Binary Grid format (aka Arc/Info Binary Grid) is the collection of the files listed above. ArcGIS treats the directory containing the files as the grid (DEM). So, when you attempt to load fg443_4443 into ArcGIS, the software sees that directory as a single DEM, not as a directory containing other files. For more information on the file format and to learn more about what each of the files above is, take a look at these links:

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=30616
http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=About_the_ESRI_grid_format
http://home.gdal.org/projects/aigrid/aigrid_format.html

Software other than ArcGIS (Global Mapper for example) does not see the whole directory as a single DEM and instead requires that you point it towards the appropriate *.adf file. In this case, you need to navigate into the fg443_4443 directory to the w001001.adf file in order for the software to load the DEM.

Finally, if you don't have access to software that will read the ESRI binary grid format, GDAL (http://www.gdal.org/) is an excellent and free tool that will allow you to convert these files into a format that is compatible with whatever software you are using.

re-post of answer to above. -cc

___________________

Good question Jim.

NCALM, who acquired the NoCal LiDAR, delivered the ESRI binary grids to us - apparently this is the format that they typically use to deliver their pre-computed DEMs to users. Given that we were under a short deadline to make these data available to the community, we did not have time to convert the data into a more generic format (such as an ascii grid). Hopefully, later releases will include the data in more generic format.

We are actively working to make the first round of NoCal point cloud data available via the GEON LiDAR Workflow and hope to have these data online in the very near future. Once the point data are online, users will be able to compute their own DEMs and output them in formats such as ascii grid.

Which GIS software package are you currently using? Although primarily an ESRI user myself, it has been my experience that many (most?) GIS software packages support the import of ESRI binary grids. If yours does not, you may consider using GDAL (http://www.gdal.org/) to convert the grids into a format that your software will read.

Please let us know if you have questions or comments. Thanks,

-cc

Originally posted by mccalpin in the old GLW forum - re-posting it here so that it is not lost - cc
______________________

Some of us use GIS systems other than ESRI's, so we cannot import LIDAR data if it is only provided in the ESRI grid format. Why don't you additionally make the data available in some non-proprietary data format, so that non-ESRI customers can import it?----Jim McCalpin