[jump-users] Better Performence for point data

Stefan Steiniger sstein at geo.uzh.ch
Fri Oct 5 14:42:46 PDT 2007


well done!

i am not sure if this makes sense, but for drawing alone one could use a 
"skip n points" mechanism or use an indexing scheme (e.g. quadtrees with 
skipping the lower(?) levels to show). I guess this would need some 
testing to identify the number of points that should be drawn maximally 
and that need to be drawn. But for the selection this is not possible 
(except the drawing). Not sure if modifying "enable check" is a solution

I guess Martin has some nice thoughts on that ;)
stefan

btw: @Martin: is you FOSS4G - DTM-TIN paper available? (I read he used 
also massive data - millions of heigh points - with postgis)

PS: have you seen this presentation on high res image browsing, which 
comes quite close to our problem, although a it is bit different domain:
	http://www.ted.com/index.php/talks/view/id/129
(sorry don't want to hijack this topic but it is related)

Larry Becker schrieb:
> I'm retesting with Arnd's new XYZ-data which contained a 62MB shape file 
> and a 33MB dbf file.
> 
> 1. OJ loaded the file in 15 seconds and finished drawing it in 55 
> seconds.  335MB committed memory.
> 
> 2.  Finished selecting about 2/3 of the points in 75 seconds Finished 
> drawing the selection handles in another two minutes.  Up to 452MB of 
> committed memory.  Right clicked on the view and it took about 2 minutes 
> to respond.
> 
> Clearly the selection of 600000+ objects is causing havoc in the UI 
> EnableChecks.  Also, the selection handle rendering optimization that I 
> made is not effective for points, so it must draw 600000 tiny yellow 
> rectangles each redraw.
> 
> So is it about memory?  I increased OJ's memory allocation to 750MB and 
> reloaded.  Same load and redraw stats.
> 
> 3.  I used the Edit->Select layer items.  It took 2 minutes to display 
> the selected items message (1000000) on the status bar.  While selection 
> handles were drawing, committed memory cycled between 650 and 700 MB.  
> Clearly, frequent pauses in drawing were caused by the JVM garbage 
> collector.  I waited for the handles to finish drawing so that I could 
> test the menu response time.  Finally, it finished.  I clicked on the 
> Edit menu.  After a few seconds the workbench window went totally blank 
> for 3 minutes, then came back.  I didn't try it again.
> 
> Conclusion:  I concur with Arnd that OpenJump is unworkable for point 
> files in excess of 500000.  The redraw of a million points is bad, 
> redraw of a million selection handles is worse, and the UI is totally 
> unresponsive with large numbers of objects selected.
> 
> I believe that we may have reached the design limits of the UI feedback 
> mechanisms.  In particular EnableCheck class methods make multiple 
> references to the number of features.  The selection feedback on the 
> status bar even computes the number of points (requiring a million 
> iterations for each access).
> 
> I repeated the experiment with JUMP 1.2 (750MB memory) which does not 
> have some of the UI enhancements like the status bar display of number 
> of selected items and points.  The first thing I noticed is that JUMP 
> renders much slower than OpenJump.  No way am I going to wait for it to 
> finish.  I did a drag select around the whole view (since there is no 
> select layer items on the Edit menu).  It is taking a long time.  I'm 
> not timing any of this since the only thing I'm interested in is the 
> menu response time.  I found that an easy test to see if the UI is 
> responsive is to mouse over a toolbar icon and see if it highlights.  
> Oops, JUMP just gave an Out of Memory Error.  End of that experiment.
> 
> I still believe the main problem is the UI checks.  The UI is totally 
> responsive until you make a large selection.  Then it gets progressively 
> slower proportional to the size of the selection.  Fixing this is the 
> number one priority.
> 
> I loaded the file in SkyJUMP which has metrics for screen redraw.  It 
> rendered the million points in 50 seconds. There is room for improvement 
> here, but this is not really the main problem.  Any optimization done to 
> optimize point drawing would also apply to selection handles.  Fixing 
> this is the number two priority.
> 
> regards,
> Larry
> 
> 
> 
> 
> 
> 
> 
> On 10/4/07, *Larry Becker* <becker.larry at gmail.com 
> <mailto:becker.larry at gmail.com>> wrote:
> 
>     OK, for my PC (P4 3.2Ghz with OJ set to use 256MB) it takes 5
>     seconds to load your 351k points u1 layer, and 17 seconds to draw. 
>     Pretty bad draw performance compared with other geometry types, but
>     not really relevant to the discussion since waiting for the redraw
>     is not necessary.
> 
>     1. I managed to copy and paste about 100k of these to a new layer in
>     about 10 seconds.  Replicate gives similar results.
> 
>     2. Selecting about 1/3 of the points takes about 5 seconds to do,
>     and about 8 more to render the handles, although as I mentioned
>     before, you can proceed as soon as the the status bar indicates the
>     number selected. 
> 
>     3. Selecting all points takes about 20 seconds, and another 30 to
>     render.
> 
>     Preliminary results for 350000 points: slow but definitely workable.
> 
>     I loaded 3 copies of the u1 layer giving a total of over 1 million
>     points.  Memory usage is at 200 MB committed.  I suspect it would be
>     quite a bit more if Stefan's data had any attributes.
> 
>     4. Redraw is now about 45 seconds, but just ignoring it works for me.
> 
>     5. Selecting about 1/3 failed after 90 seconds and resulted in an
>     Out of Memory Error.
> 
>     After allocating 512MB of memory to OJ, I restarted.  390000 points
>     selected after 20 seconds.  The selection handles finished rendering
>     40 seconds later. 388MB in use.
> 
>     6. Right clicking on the selection takes about 5 seconds to bring up
>     the menu.
> 
>     7.  Replicate took about 20 seconds to replicate to a new layer.
> 
>     These are my current findings.  I'll update the post when I test
>     with Arnd's data.  So far I don't see anything too bad.  About what
>     you might expect really.  The rendering code is not optimized for
>     points, and the selection code is dealing with its worst case since
>     the bounding box for points is a point.
> 
>     Larry
> 
>     On 10/4/07, *Stefan Steiniger* < sstein at geo.uzh.ch
>     <mailto:sstein at geo.uzh.ch>> wrote:
> 
>         oha..
>         that is interesting. Accidentaly i stored the points as multipoints.
>         Now i updated my jump @office and stored the points as point
>         (and wrote
>         the file again to the ftp). The file size is now already a bit
>         smaller,
>         and also the commited memory...
> 
>         stefan
> 
>         Stefan Steiniger wrote:
> 
>         >  Good if Arnd provides the data, but here the link to my data
>         in case
>         >  it fails:
>         >  ftp://ftp.geo.uzh.ch/pub/sstein/data/u1_points.zip
>         >
>         >  an option to get one million points is to copy the 350k points and
>         >  then to move them (if that works ;).
>         >  Btw. the lascers can data have no attributes (may be therefore
>         smaller
>         >  than Arnd's; the covered area is relatively small < 200m)
>         >  (shapefile size is 30mb and zip xyz-ascii was 15mb)
>         >
>         >  stefan
>         >
>         >  Arnd Kielhorn wrote:
>         >
>         > > Hello Larry,
>         > >
>         > > first of all, thank You very much!
>         > >
>         > > I work with delaunay triangulation and building a 3D-Model
>         with our
>         > > plugin.
>         > >
>         > > If we both could make a direct connection (via FTP or a free
>         tool
>         > > called teamviewer) I could sent You one file with
>         laserscanning data
>         > > with 1 million points (XYZ). It is about 30 MB in the PIROL-CSV
>         > > format or about 100 MB as shape file.
>         > >
>         > > Kindly regards
>         > > Arnd
>         > >
>         > > Larry Becker schrieb:
>         > >
>         > >> I'm going to start investigating the point performance
>         problem as
>         > >> soon as I get some test data.  I have been trying to locate
>         a large
>         > >> point dataset on the web, but no luck.  Perhaps the best
>         approach
>         > >> might be to add a new item to the Generate menu, "Random
>         Points".
>         > >>
>         > >> @Arnd: One thing that people often forget about JUMP is that
>         > >> rendering is a background  process.  You don't have to wait
>         for it
>         > >> to finish to start the next operation.
>         > >> What kind of terrain maps are you generating?  Are you
>         generating
>         > >> contour lines from a grid of points?  You mentioned deleting
>         > >> selected points being impossibly slow.  I don't see why this
>         should
>         > >> be so.  If I can duplicate the problem, it should be an easy
>         fix.
>         > >>
>         > >> regards,
>         > >> Larry
>         > >>
>         > >> On 10/2/07, *Larry Becker* < becker.larry at gmail.com
>         <mailto:becker.larry at gmail.com>
>         > >> <mailto: becker.larry at gmail.com
>         <mailto:becker.larry at gmail.com>>> wrote:
>         > >>
>         > >>     HI Arnd,
>         > >>
>         > >>       Would it be helpful to work around the problem by
>         saving the
>         > >>     points as a shapefile and using a utility like shp2tile
>         to break
>         > >>     the point data up into multiple files?
>         > >>
>         > >>     see: http://imaptools.com/download-software.html
>         > >>
>         > >>     regards,
>         > >>     Larry
>         > >>
>         > >>
>         > >>     On 10/2/07, *Arnd Kielhorn* < a.kielhorn at gmx.de
>         <mailto:a.kielhorn at gmx.de>
>         > >>     <mailto:a.kielhorn at gmx.de <mailto:a.kielhorn at gmx.de>>>
>         wrote:
>         > >>
>         > >>         Hello Paul,
>         > >>
>         > >>         thank You for the tip with the z-value, but how I
>         can use it
>         > >> as a
>         > >>         z-value on the point itself?
>         > >>
>         > >>         The file fomrat is txt, csv (from PIROL CSV-Format) like
>         > >>         following example
>         > >>         $Rechts    Hoch    heigh
>         > >>         $grad    grad    m
>         > >>         $double    double    double
>         > >>         3434000.00    5800000.00     93.78
>         > >>         3434001.00    5800000.00    93.84
>         > >>         ...   ...   ...
>         > >>
>         > >>         or the ESRI ascii grid format asc (reading plugin
>         from SIGLE).
>         > >>         The most files I got have 1 million points and such
>         csv files
>         > >>         have about
>         > >>         30 MB.
>         > >>         In my start file for OJ I got 768 MB, because I have
>         1 GB RAM.
>         > >>
>         > >>         Kindly regards
>         > >>         Arnd
>         > >>
>         > >>
>         > >>         Paul Austin schrieb:
>         > >>         > Arnd,
>         > >>         >
>         > >>         > What file format are you using?
>         > >>         >
>         > >>         > As JUMP loads the whole file into memory I would
>         recommend
>         > >>         setting the
>         > >>         > heap size on the JAVA VM to be -Xmx512M if you are
>         dealing
>         > >>         with large
>         > >>         > datasets.
>         > >>         >
>         > >>         > You mentioned that the height is an attribute on
>         the feature,
>         > >>         if you use
>         > >>         > the height as a z-value on the point itself there
>         will be
>         > >>         some memory
>         > >>         > savings.
>         > >>         >
>         > >>         > Paul
>         > >>         >
>         > >>         > Arnd Kielhorn wrote:
>         > >>         >
>         > >>         >> Hello Larry,
>         > >>         >>
>         > >>         >> as in the further discussion just named the main
>         problema
>         > >>         are (for
>         > >>         >> example I just have to work with point layers
>         with only a
>         > >> heigh
>         > >>         >> attribute but with 1 million points; but also
>         e.g. 300,000
>         > >>         points are
>         > >>         >> enough):
>         > >>         >> - (re)drawing of the points
>         > >>         >> - strongly restricted or impossible operation
>         (e.g. deleting
>         > >>         some
>         > >>         >> points/vertices after marking them; e.g. copying
>         hundreds or
>         > >>         a few
>         > >>         >> thousand in a new layer to have layer with lesser
>         points)
>         > >>         >> Very often OJ hangs on and I have to restart it.
>         > >>         >> I use such point layers for creating a digital
>         terrain maps.
>         > >>         >>
>         > >>         >> Kindly regards
>         > >>         >> Arnd
>         > >>         >>
>         > >>         >> Larry Becker schrieb:
>         > >>         >>
>         > >>         >>> Hi Arnd,
>         > >>         >>>
>         > >>         >>>   I haven't worked with large point datasets
>         much.  What
>         > >> it is
>         > >>         >>> exactly that is slow?  Is it redraw time?  Or
>         perhaps the
>         > >>         particular
>         > >>         >>> operation that you are doing?
>         > >>         >>>
>         > >>         >>>   I know that point layers do not benefit from
>         the recent
>         > >>         decimation
>         > >>         >>> optimization, so they take longer to draw than
>         equivalent
>         > >>         polygons
>         > >>         >>> and polyline layers.  I think Michaël tried a point
>         > >> decimation
>         > >>         >>> optimization, but wasn't satisfied with the results.
>         > >>         >>>
>         > >>         >>> regards,
>         > >>         >>> Larry Becker
>         > >>         >>>
>         > >>         >>> On 10/1/07, *Stefan Steiniger* <
>         sstein at geo.uzh.ch <mailto:sstein at geo.uzh.ch>
>         > >>         <mailto: sstein at geo.uzh.ch <mailto:sstein at geo.uzh.ch>>
>         > >>         >>> <mailto: sstein at geo.uzh.ch
>         <mailto:sstein at geo.uzh.ch> <mailto: sstein at geo.uzh.ch
>         <mailto:sstein at geo.uzh.ch>>>>
>         > >> wrote:
>         > >>         >>>
>         > >>         >>>     mhm
>         > >>         >>>     from my point of view it is both OJ and JRE.
>         It would
>         > >>         be probably
>         > >>         >>>     better
>         > >>         >>>     when the points are stored and processes in
>         a database.
>         > >>         >>>
>         > >>         >>>     stefan
>         > >>         >>>
>         > >>         >>>     Arnd Kielhorn schrieb:
>         > >>         >>>     > Hello,
>         > >>         >>>     >
>         > >>         >>>     > yes I know, first of all OJ is a GIS for
>         vector data.
>         > >>         But the
>         > >>         >>>     > functionality is also very good for point
>         datasets.
>         > >>         But working
>         > >>         >>> with
>         > >>         >>>     > great point datasets (more 500,000 points)
>         bring
>         > >>         problems with the
>         > >>         >>>     > memory and the garbage collector works not
>         satisfied.
>         > >>         Every
>         > >>         >>>     command take
>         > >>         >>>     > a lot of time or could not carried out.
>         (CPU P4 2.4
>         > >>         GHz, 1 GB RAM)
>         > >>         >>>     > Is it only a thing of the JRE or also from
>         OJ?
>         > >>         >>>     > So, it is useful to know if there is a
>         chance to
>         > >>         increase the
>         > >>         >>>     > performence of OJ in this case and what
>         could be the
>         > >>         possible
>         > >>         >>>     milestones
>         > >>         >>>     > to reach this aim?
>         > >>         >>>     > I am very eager to read everyones opion.
>         > >>         >>>     >
>         > >>         >>>     > Kindly regards
>         > >>         >>>     > Arnd
>         > >>         >>>     >
>         _______________________________________________
>         > >>         >>>     > jump-users mailing list
>         > >>         >>>     > jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>
>         > >>         <mailto: jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>>
>         > >>         >>>     <mailto: jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>
>         > >>         <mailto:jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>>>
>         > >>         >>>     >
>         > >>         http://lists.refractions.net/mailman/listinfo/jump-users
>         > >>         <
>         http://lists.refractions.net/mailman/listinfo/jump-users>
>         > >>         >>>     >
>         > >>
>         > >> <http://amusingprogrammer.blogspot.com/
>         <http://amusingprogrammer.blogspot.com/>>
>         > >>
>         ------------------------------------------------------------------------
> 
>         > >>
>         > >>
>         > >> _______________________________________________
>         > >> jump-users mailing list
>         > >> jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>
>         > >> http://lists.refractions.net/mailman/listinfo/jump-users
>         > >>
>         > >
>         > >
>         > >
>         > > _______________________________________________
>         > > jump-users mailing list
>         > > jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>
>         > > http://lists.refractions.net/mailman/listinfo/jump-users
>         <http://lists.refractions.net/mailman/listinfo/jump-users>
>         > >
>         > >
>         >
>         >  _______________________________________________
>         >  jump-users mailing list
>         >  jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>
>         >  http://lists.refractions.net/mailman/listinfo/jump-users
>         >
>         >
> 
>         _______________________________________________
>         jump-users mailing list
>         jump-users at lists.jump-project.org
>         <mailto:jump-users at lists.jump-project.org>
>         http://lists.refractions.net/mailman/listinfo/jump-users
> 
> 
> 
> 
>     -- 
>     http://amusingprogrammer.blogspot.com/
>     <http://amusingprogrammer.blogspot.com/> 
> 
> 
> 
> 
> -- 
> http://amusingprogrammer.blogspot.com/
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> jump-users mailing list
> jump-users at lists.jump-project.org
> http://lists.refractions.net/mailman/listinfo/jump-users


More information about the jump-users mailing list