It is currently Fri, 20-10-17, 1:11 GMT

All times are UTC




Post new topic Reply to topic  [ 196 posts ]  Go to page 1, 2, 3, 4, 5 ... 14  Next
Author Message
PostPosted: Tue, 04-09-07, 18:32 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4496
Location: Hamburg, Germany
Testing the code of the tools and locating potential bugs is a crucial task! A further important activity is to think about various optimizations that will further boost the performance of the tools!

Looking forward to your participation!

Bye Fridger

_________________
Image


Last edited by t00fri on Fri, 07-09-07, 9:02 GMT, edited 2 times in total.

Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 9:56 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 2:32 GMT
Posts: 430
Location: South Korea
Ok, I've successfully compiled F-TexTools 1.0pre1 for Mac OS X (had to edit the Makefile - I'll contribute back my changes once I'm done testing).

Testing wise, the 85k BMNG took all afternoon on my old machine.
These are the same tests that Fridger performed on a 3.2GHz P4 with 3G ram..
My machine is a 1.33GHz laptop with 1G ram running Mac OS X. Files were written to an external 120G HDD. Only 256MB total swap space was allocated and used by the system.

Here are some benchmarks:

1) Reduce the binary input file of width = 86400 pixels to the nearest power-of-two dimension (65536 x 32768); then reduce it to only 16384 pixels and convert it to PNG output format (PNG_compression = 6).
Code:
gzip -dc < <RGB>.bin.gz | tx2pow2 3 86400 | tx2half 3 65536 | tx2half 3 32768 | bin2png 3 16384  >  <outfile>.png

Fridger: 11.5 min
Me: about 1 hour (almost 6 times as slow - BTW my processor is 1/3 the speed and memory is 1/3)
Confirmed that result is a valid PNG by opening it in GIMP.

2) Generate a much improved specmap in 1 x 8 bit raw format of width = 65536 pixels from the official BMNG watermask and the 16 bit elevation graymap with recommended residual light level = 0.12 for the land areas.
Code:
gzip -dc < <elevation_map>.bin.gz | specmap 86400 <watermask>.bin 0.12 | tx2pow2 1 86400 > <specmap>.bin

Fridger: 4.1 min
Me: about 30 mins (almost 7.5 times as slow)

3) Read a 3x8 bit gzipped RGB base texture of width = 86400 pixels from STDIN and mount the improved specmap texture of the same width as alpha channel of the resulting RGBA map. Subsequently, generate a set of 2048 optimized RGBA VT tiles of level 5 in PNG format.
Code:
   
gzip -dc < <RGB>.bin.gz | tx2rgba 86400 <specmap>.bin | tx2pow2 4 86400 | txtiles 4 65536 5

Fridger: 16.6 min
Me: 99 mins 10 seconds (about 6 times as slow)
Confirmed 2048 tx*.png files created.


I guess there are two factors limiting performance here - cpu clock, and my external HDD's usb link speed.
It's usb 2.0, and it's quite fast when connected to my Windows machine but my laptop's usb speed is not stellar.
Memory is probably not an issue - I have 588MB real memory free while running test 3.


Last edited by dirkpitt on Sat, 15-09-07, 10:49 GMT, edited 2 times in total.

Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 10:04 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4496
Location: Hamburg, Germany
DW,

many thanks for your first report. In my experience these differences are strongly related to the compile options used, including libpng! For example, I used the assembler version for libpng (also in Windows, where I compiled it myself) . The ASM version alone brings >30% speed relative to the normal one.

I'll also quote the analogous times under Windows with my Core 2 Duo, 2GHz notebook and 2 GB of RAM later today.


Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 10:23 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 2:32 GMT
Posts: 430
Location: South Korea
There are no equivalent assembler optimizations for the Mac PPC version of libpng so I could imagine some speed being lost there. BTW, assembler optimizations are being discontinued from the most recent versions of libpng! (But I'm using 1.2.8)


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 10:55 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 2:32 GMT
Posts: 430
Location: South Korea
Finally test 3 finished - as expected it took 99 minutes.
However the alpha channel looks messed up - I suspect I needed to do a byteswap when generating the specmap.
This is weird, because the BMNG readme claims that the srtm dataset is big-endian - the same as my machine. What's going on here?!
Fridger, is the srtm dataset little-endian?


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 11:24 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4496
Location: Hamburg, Germany
DW,

just returned from our weekly shopping... Here (GMT+2h) it's now 13:23 o'clock.

  • First of all:
    Please tick the Disable HTML in this post option in the editor window, otherwise in commandline displays a number of pieces get lost/are swallowed. Have a look in yours! Some characters are interpreted as HTML tokens, erroneously...
  • I suppose you compiled with gcc. What gcc optimizations did you use, precisely? A factor 6-7.5 slower is really dramatic! My former PhD student also has an apple powerbook which --despite its nice design-- I find painfully slow... But still that factor is too big. And it also cannot be a memory issue since very little memory is used during the processing. My Desktop (Linux) is > 3 years old already.
  • OK I will now do the tests with my Core 2 Duo 2.0 GHz notebook under Windows.
  • Incidentally I always test the time as
    Code:

    time bash -c '<commandline>'


    where <commandline> stands for the entire commandline with pipe modules and STDIN/STDOUT attachements. My times referred to the 'user' fraction of the quoted times.


Local time in Tokyo is 20:23 right now, correct?

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 11:31 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4496
Location: Hamburg, Germany
dirkpitt wrote:
Finally test 3 finished - as expected it took 99 minutes.
However the alpha channel looks messed up - I suspect I needed to do a byteswap when generating the specmap.
This is weird, because the BMNG readme claims that the srtm dataset is big-endian - the same as my machine. What's going on here?!
Fridger, is the srtm dataset little-endian?


My computer is definitely 'little-endian' in agreement with what the output of specmap says. The srtm2 dataset is definitely 'big-endian'. Correspondingly, I needed a byteswap =1 parameter in tx2pow2 as you could see in my command line above. So you should have omitted the byteswap in tx2pow2.

Did you do that?

Cheers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 11:45 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4496
Location: Hamburg, Germany
DW,

in order to trace that speed issue with your MAC better, it might be useful to do some separate timing with commandline tools that DO NOT involve libpng.

So it would be sensible to redo and compare my first example without the FINAL PNG conversion on both our machines!

-F.

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 12:21 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 2:32 GMT
Posts: 430
Location: South Korea
t00fri wrote:
[*] Incidentally I always test the time as
Code:

time bash -c '<commandline>'


where <commandline> stands for the entire commandline with pipe modules and STDIN/STDOUT attachements. My times referred to the 'user' fraction of the quoted times.
[/list]

Local time in Tokyo is 20:23 right now, correct?


Something like that yes (now it's 21:15, I need to grab a bite to eat).
I actually did use the time command, but my times are the "real" (total time) time, i.e., user+sys. Now that I think about it though, the sys overhead is probably not very relevant when benchmarking the apps.

Anyway, I'll try and redo without the byteswap, and leaving out the bin2png / txtiles steps. Will post once it's done (may take almost 2 hours...)


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 12:29 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 2:32 GMT
Posts: 430
Location: South Korea
Wait a moment, the only tool with a byte swap option is specmap, and it's "don't swap" by default.
My command line didn't specify the byte swap flag, so it shouldn't have swapped.. will check
whether this is a specmap bug.


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 15-09-07, 22:21 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4496
Location: Hamburg, Germany
DW,

after quite extensive testing today, I also located the apparent ;-) problem with the scrambled alpha channel in the resulting level 5 tiles of example 3). It is NOT a byteswap problem and nothing was wrong in my code!

It was really my fault when composing the three commandline examples 1) - 3) in the README of the F-TexTools-1.0pre1 archive:

Example 2)
was to generate the improved specmap.bin file
Code:
gzip -dc < <elevation_map>.bin.gz | specmap 86400 <watermask>.bin 0.12 | tx2pow2 1 86400 > <specmap>.bin


and you presumably also used it for the subsequent calculation of the 2048 tiles, the alpha channel of which appeared scrambled:
Example 3)
Code:
gzip -dc < <RGB-infile>.bin.gz | tx2rgba 86400 <specmap>.bin | tx2pow2 4 86400 | txtiles 4 65536 5


------------------------------------------------------------------------------
The catch was that in example 2) the /last/ pipe command "|tx2pow2 1 86400" reduced the specmap.bin file to a width=65536 pixels!!! Hence it could NOT be used in Example 3) where a specmap.bin file of width=86400 pixels is needed! Using the smaller one, caused the scrambled alpha channel in the end!
-------------------------------------------------------------------------------

So please redo Example 2) as follows:

Code:
gzip -dc < srtm_ramp2.world.86400x43200.bin.gz | specmap 86400 world.watermask.86400x43200.bin  0.12 > specmap.bin


Of course, I needed an additional byteswap =1 , since srtm_ramp2.world.86400x43200.bin is 'big-endian' and my PC is 'little-endian' (unlike your MAC).

Then you may use the resulting specmap.bin file in example 3) and the final VT-tiles come out perfect.

Incidentally:

If you want to test the byteswap mechanism further, you probably know that you can easily obtain byteswapped bin files with the UNIX command:

dd if=<infile>.bin of=<outfile>.bin conv=swab

---------------------------------------------------------------------------
Last not least: the times that my Core 2 Duo 2.GHz DELL notebook/ 2 GB RAM takes for doing examples 1) - 3) under Windows XP are quite close to the times my 3.2 GHz P4/3 GB RAM Desktop needs for the identical task under Linux.

Hence it remains an important task to find out why your MAC performs so much slower ( factor 6 - 7.5)!!??
---------------------------------------------------------------------------

Cheers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 16-09-07, 10:21 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 2:32 GMT
Posts: 430
Location: South Korea
Bummer.. my power adapter for my Mac stopped working today.
Actually, it looks really bad - the connection between the wire and the plug has turned green - like it's started to burn or something :shock:
I'll have to go out and buy a new one. Luckily I spotted the burn before it turned into a bonfire.


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 16-09-07, 14:38 GMT 
Offline
User avatar

Joined: Mon, 03-09-07, 23:01 GMT
Posts: 388
Location: Tuscany, Tyrrhenian Sea
dirkpitt wrote:
Bummer.. my power adapter for my Mac stopped working today.
Actually, it looks really bad - the connection between the wire and the plug has turned green - like it's started to burn or something :shock:
I'll have to go out and buy a new one. Luckily I spotted the burn before it turned into a bonfire.


Ugh! Fridger, your tools are very powerful; at end of result, a computer fire up! (I'm joking, of course).

_________________
Never at rest.


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 16-09-07, 14:49 GMT 
Offline
User avatar

Joined: Mon, 03-09-07, 23:01 GMT
Posts: 388
Location: Tuscany, Tyrrhenian Sea
Dirkpitt, do you have an UPC between the Mac and the power source? It would protect from things as these.

_________________
Never at rest.


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 16-09-07, 21:27 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 2:32 GMT
Posts: 430
Location: South Korea
I'm back, and armed with a new power adapter I once more "fired up" F-TexTools.
Looks like I'm not the only person "burned" by an adapter. For example look at this guy's:
http://www.neowin.net/forum/index.php?showtopic=531840

A UPC at home? Somehow I think a fire extinguisher is going to be more useful in my case.. :wink:


Anyway, back to work.
I ran test 2 and 3 again. This time the tiles look perfect, and here are the time results:

Test 2 (generate specmap): 26 min 11 seconds (previous result: 30 mins)
=> Fridger * 6.3
Test 3 (generate 2048 png tiles): 88 min 30 seconds (previous result: 99 mins)
=> Fridger * 5.3

So proof that a new power adapter upgrades my computer's speed? :shock:
Of course not! What probably happened, was that my ex-adapter, before it burned out,
was intermittently failing. My laptop - like many others - reduces processor speed when not
running off AC. If the adapter doesn't work properly, the laptop switches to battery mode,
hence the poor benchmarks previously.

Even so, 6x and 5x slower than Fridger's results is not so "hot". :wink: I think the reason,
which I hinted at before, may be because of my slow HD link speed. I'll try and benchmark
by omitting the png steps, later today.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 196 posts ]  Go to page 1, 2, 3, 4, 5 ... 14  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group