<div>Hi,</div>
<div>I also saw many times this kind of posts, and I think your comments of today are for the first time a positive contribution. As a matter of fact I never thought a raster in the way you describe here, but now I discover that in the past I would love to manage my scanned maps in this way!!. Also I recall that I used to get the number 2 point form your post but using ECW (Encapsulated Compressed Wavelet) and an API provided by ERMapper (I think it was free). Now I have to check but maybe ERMapper also have something for JPG2000. Is it bad idea to use ECW format?.
</div>
<div><br>Mario Servin<br> </div>
<div><span class="gmail_quote">On 10/24/06, <b class="gmail_sendername">Marshall, Steve</b> <<a href="mailto:smarshall@wsi.com">smarshall@wsi.com</a>> wrote:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>
<div><font face="Arial" size="2"><span>Over the past several months, I've seen a number of postings regarding support in PostGIS for raster data. The email threads tend to look like this:</span></font></div>
<div><font face="Arial" size="2"><span>---</span></font></div>
<div><font face="Arial" size="2"><span>Poster: </span></font></div>
<div><font face="Arial" size="2"><span>I'd like to have support for raster data in PostGIS.</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>Responder: </span></font></div>
<div><font face="Arial" size="2"><span>If you are just going to put the whole image in the database, then take the complete image back out, what's the point. Why do you think storing in the RDB would be better than managing raster data in files?
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>Poster: </span></font></div>
<div><font face="Arial" size="2"><span>I would not have to manage the interactions between the RDB and a file-based database, the images would follow transactional semantics, and the raster data would be network accessible. Additionally, I could use PostGIS spatial queries to relate vector data to raster data, and maybe even write some additional raster manipulation functions, all of which would be built into the database.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>Responder:</span></font></div>
<div><font face="Arial" size="2"><span>That stuff sounds neat, but the performance issues are unworkable. Performance with large images will be terrible if we use anything other than flat files.</span></font></div>
<div><font face="Arial" size="2"><span>---</span></font></div>
<div><font face="Arial" size="2"><span>Generally, the debate ends here. This time I'd like to see if the conversation could go in another direction.</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>I think the "what's the point" response is basically valid for a naive implementation of images within a RDB. If you are going to treat an image as a blob that can be extracted only in the same form that it was inserted, there is really not much point in using relational storage. The raster data will need to be read into program memory buffers in one big chunk. For large images, the data will likely need to be written into a temporary file so it can be incrementally processed using file seeks. In this case, you'd be better off having the data in a file in the first place.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>However, if you support multiple modes of extraction, then the relational model really starts to become compelling. For example, I could see extracting an image that is a subsector of the complete image (
i.e. a smaller image that covers a smaller geographic region). One could also imagine extracting a low-resolution version of the image for "zoomed-out" display. Low-res images would cover the same geographic area, but with fewer pixels, so less processing for applications. By extracting image subsets, large image processing could be done straight from the database, without bloating application memory or using an intermediate temporary file.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>The ability to do both resolution and sector types of subsetting are well-supported by the JPEG2000 image format. In fact, these types of operations were some of the drivers behind the JPEG2000 design. Given that JPEG2000 compresses well and is an open standard, it seems like a good format for storing raster data. So is there some way to store JPEG2000 data in an intelligent way within a relational database?
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>Essentially, I think you could store JPEG2000 in a database by defining a new data type (e.g. pgraster) that could hold the image data in JPEG2000 format. The data type might also populate some derived data when the object is written by interpreting the JPEG2000, such as a PostGIS GEOMETRY object to represent the bounding box of the raster. This derived data would allow more optimal responses to spatial queries, but overall would not be mandatory to the implementation. However, it would be mandatory to define a set of SQL functions that take a pgraster argument to allow one to extract different subsets of the JPEG2000 data.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>In terms of implementation, I think the pgraster implementation would require the following:</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>1. The underlying database storage would allow random access into the image data.</span></font></div>
<div><font face="Arial" size="2"><span>2. A fairly sophisticated JPEG2000 codec would need to be linked into the database server, so that different subsets of the data could be accessed.</span></font></div>
<div><font face="Arial" size="2"><span>3. The JPEG2000 codec would need to be integrated with the RDB storage so that one could use standard codec functions with a RDB storage model.</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>I think #1 could be achieved by making the pgraster type "toastable", and using a PostgreSQL TOAST table for the underlying storage. The data would not use TOAST compression, since the image should already be well-compressed. We could use the internal PostgreSQL function heap_tuple_untoast_attr_slice() to extract subsets of the toasted data, so we do not need to detoast the entire image during processing. Toast does not provide an API for a similar kind of seeking during writes, but I think it's the seeking on reads that will be the most significant to performance.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>I think #2 is potentially more problematic. The publicly available JPEG2000 codecs do not seem to have the interfaces needed for extraction of parts of a JPEG2000 image. The JasPer library only provides the encode and decode functions that produce or accept a jas_image_t type; it doesn't have any of partial extraction capabilities. The Open JPEG codec is a bit tougher to get a handle on, but it also appears to only allow translation between an image type and a codestream.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>The code that really seems well-adapted to this problem is the Kakadu package written by David Taubman, one of the originators of JPEG 2000. Unfortunately, this library is not open source. Kakadu has been included as an optional component of other open source projects, like GDAL. However, I think Kakadu's license would come into conflict with the GPL used by PostGIS. I'm not a lawyer, but I think this conflict could be overcome if PostGIS could be release under dual licenses, such as GPL or LGPL. Whether or not its desirable to include of Kakadu in a PostGIS extension is another question.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>If anyone has knowledge of other JPEG 2000 codecs that have these low-level access capabilities, I'd be very happy to hear about them. Also, if I've mischaracterized any of the codecs, I'd love to be corrected.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>In any event, I'm curious to see if there is significant interest in an implementation of JPEG2000 raster data type within PostGIS. If so, I think I could dedicate a significant amount of my time over the next several months, as well as perhaps some funding from my employer, depending upon whether some of the issues I raised above can be resolved.
</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div>
<div><font face="Arial" size="2"><span>Steve Marshall</span></font></div>
<div><font face="Arial" size="2"><span></span></font> </div></div><br>_______________________________________________<br>postgis-users mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:postgis-users@postgis.refractions.net">
postgis-users@postgis.refractions.net</a><br><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://postgis.refractions.net/mailman/listinfo/postgis-users" target="_blank">http://postgis.refractions.net/mailman/listinfo/postgis-users
</a><br><br><br></blockquote></div><br>