It is currently Mon, 18-02-19, 4:21 GMT

All times are UTC




Post new topic Reply to topic  [ 33 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject:
PostPosted: Tue, 18-09-07, 20:25 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
Hi all,

I prepared this highly reduced 5400x2700 scientific BMNG data archive to stimulate QUICK practising with typical F-TexTools console commands and the respective binary scientific data files!

5400x2700 'dwarf'-testfiles for F-TexTools

The following three command-line examples for my new F-TexTools are identical to the original ones in the archive's README, except that they are specialized to "tiny" 5400x2700 file sizes that can be VERY quickly downloaded and worked upon!

Once you know how to use the tools with these small files, you will have accumulated enough curiosity, self-confidence and patience to go for the really BIG 85k stuff ;-). I promise the results are worth trying!

The Three Examples
==============
Code:
 
1) gzip -dc < world.200406.3x5400x2700.bin.gz | tx2pow2 3 5400 | tx2half 3 4096 | tx2half 3 2048 | bin2png 3 1024  >  outfile1k.png

2) gzip -dc < srtm_ramp2.world.5400x2700.bin.gz | specmap 5400 world.watermask.5400x2700.bin 0.12 1 > specmap.bin

3) gzip -dc < world.200406.3x5400x2700.bin.gz | tx2rgba 5400 specmap.bin | tx2pow2 4 5400 | txtiles 4 4096 1


Like the 16 times (!) wider 85k original, the 5400x2700 'dwarf'-sized textures are NOT a power-of-two size. That's why 'tx2pow2' is called for to reduce them to standard 4096x2048 size. In exercise 3) we generate only 8 RGBA tiles (1k x 1k, level 1), instead of 2048 RGBA tiles with the 85k original texture! If you display the tiles with xnview, you can toggle the alpha channel mask <=> specular texture by hitting the M key! Check it...

All these exercises now only take SECONDS, so it's really worth trying for yourself!

------------------------------------------------------
The 16bit 'dwarf'-elevation map is stored in 'big-endian' mode, just like it's "big brother". So with usual 'little-endian' PC's, you need to apply the byteswap=1 option in example 2) as shown above. For 'big-endian' computers, like PowerPC MAC's NO byteswap is needed, however!
-------------------------------------------------------

Enjoy and let me know about your experiences,

Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 24-09-07, 20:12 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
Let me inform people that on the NVIDIA developer site, there is now a new Forum about the new NVIDIA texture tools (moderated by Ignacio Castano). Here is the link:

http://developer.nvidia.com/forums/index.php

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 30-09-07, 15:42 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
Hi all,

I am pleased to announce the second update of the nmtools, for making the greatest monster normalmaps for Celestia directly from the scientific imaging data:

NmTools-1.5pre1

along with a binary installer version for Windows users:

NmTools-1.5pre1.bin.exe

which --as usual-- can be just installed by double-clicking the archive. Again, a registry entry is made such that the tools can be executed from every directory by just calling their names from the console. You may have to reboot after installation for this to work. Also don't forget to deinstall the previous version (1.0.1) of the nmtools!

The first, above archive contains besides the sources, Makefiles and VS-2003.net project files all the binaries for Windows, MAC-OSX-Universal (PPC and Intel) and Linux!

Here are the release highlights from the ChangeLog file
  • created a ChangeLog file and started logging the changes
  • created .doc versions of README and Changelog for Windows users
  • replaced the 16 bit IO by considerably faster code
  • renaming of the individual tools to match the naming systematics of the
    F-TexTools. The latter serve to making the remaining Celestia textures
    in highest quality on normal home computers.
  • replaced standard floor() and nint() rounding routines by much faster ones,
    using "magic numbers" based on Sree Kotay's xs_RoundToInt() and
    xs_FloorToInt(), http://www.stereopsis.com/sree/fpu2006.html
  • made it work also for Linux, using the fix from
    http://www.wrcad.com/linux_numerics.txt
  • replaced the original single pass resize_to_power_of_two algorithm with a
    faster double pass one.
  • the overall speedup amounts to about a factor of TWO!
  • In all cases the tool nvimgdiff from the new nvidia-texture tools was
    instrumental to assert that the generated images were /identical/ to the
    original ones after implementation of the speed-up code.
  • the VT-tiles generation tool now directly produces PNG formatted tiles
    (instead of PPM format)
    ! This allows their immediate use in Celestia or
    further packing into the high-quality DXT5nm formatted tiles by means of
    the new OpenSource nvidia-texture-tools.
  • overhauled considerably the output logging during operation of the tools.
For new users I STRONGLY recommend my detailed tutorial, where all pecularities about installation, endedness and command-line syntax are explained step-by-step with many examples. Don't forget the renaming of the tools in this release, however.

Let me know about
  • how it goes...
  • your individual speed increase, please!


Throughout, I want to cordially acknowledge active help and good ideas by

Da Woon Jung <dirkpitt2050@users.sf.net> aka dirkpitt aka DW
Christophe Teyssier <chris@teyssier.org>

Enjoy,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 30-09-07, 16:29 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
Hi all,

here is a couple of quickstart examples for testing the new NmTools-update above, using the 'dwarf' data set which I have uploaded recently,

5400x2700 'dwarf'-testfiles for the F-TexTools

It is precisely for quick testing and "warming up"!

Download the data and change to that data directory, using the console (or e.g. a cygwin terminal):
Type (for a little-endian Intel-PC or Intel-MAC):

Code:
gzip -cd < srtm_ramp2.world.5400x2700.bin.gz| nm2pow2 5400 1 | nms 6378.140 4096 5.0 > nm4k.png

For Big-endian PPC-MAC's type instead (see my tutorial!)
Code:
gzip -cd < srtm_ramp2.world.5400x2700.bin.gz| nm2pow2 5400 | nms 6378.140 4096 5.0 > nm4k.png

Inspect the resulting exaggerated 4k normalmap (nm4k.png) with your favorite image viewer!
On my 3.2 GHz Desktop, this just took 5 seconds and the output looks like so

Code:
> time bash -c 'gzip -cd < srtm_ramp2.world.5400x2700.bin.gz| nm2pow2 5400 1 | nms 6378.140 4096 5.0 > nm4k.png'

[nm2pow2]: Reducing to nearest power-of-two size:  4096 x  2048
[nms    ]: Input file is a  16 bit elevation map:  4096 x  2048

           Generating a normalmap for spherical geometry in PNG format

pow2[  1024 rows of 2048 ->   0.11 s]
nms [  1024 rows of 2048 ->   1.66 s]
pow2[  2048 rows of 2048 ->   0.23 s]
nms [  2048 rows of 2048 ->   3.08 s]

bash -c   3.50s user 0.16s system 72% cpu 5.024 total


Note, the additional embedding

Code:
time bash -c '<commandline>'


works for MAC's, Linux and CYGWIN/Windows, and is a precise way of timing the commandline execution!


Here is another example (specialized) for little-endian users:

Code:
> time bash -c 'gzip -cd < srtm_ramp2.world.5400x2700.bin.gz| nm2pow2 5400 1 | nmtiles 1 6378.140 4096 5.0'

[nm2pow2]: Reducing to nearest power-of-two size:  4096 x  2048
[nmtiles]: Input file is a  16 bit elevation map:  4096 x  2048

           Generating  8 optimized normalmap tiles for level 1
           in PNG format, with sizes from 1024 x 1024 to 1024 x 1024

pow2[  1024 rows of 2048 ->   0.16 s]
tile[    4 VT's of     8 ->   1.27 s]
pow2[  2048 rows of 2048 ->   0.27 s]
tile[    8 VT's of     8 ->   2.35 s]

bash -c   2.82s user 0.12s system 99% cpu 2.963 total


In just 2.9 secs, 8 (1024x1024)-normalmap tiles in PNG format are generated!

Enjoy,
Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Tue, 02-10-07, 18:51 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
Hi all,

I am pleased to announce the third update of the F-TexTools, for making the greatest monster textures for Celestia, directly from the scientific imaging data:

F-TexTools-1.0pre3

along with a binary installer version for Windows users:

F-TexTools-1.0pre3.bin.exe

which --as usual-- can be just installed by double-clicking the archive. Again, a registry entry is made such that the tools can be executed from every directory by just calling their names from the console. You may have to reboot after installation for this to work. Also don't forget to deinstall the previous version (F-TexTools-1.0pre2.bin.exe)!

The first, above archive contains besides the sources, Makefiles and VS-2003.net project files, all the binaries for Windows, MAC-OSX-Universal (PPC and Intel) and Linux!

Here are the release highlights from the ChangeLog file
  • replaced standard floor() and nint() rounding routines by much faster ones,
    using "magic numbers" based on Sree Kotay's xs_RoundToInt() and xs_FloorToInt(), http://www.stereopsis.com/sree/fpu2006.html
  • made it work also for Linux, using the fix from http://www.wrcad.com/linux_numerics.txt
  • replaced the original single pass resize_to_power_of_two algorithm with a faster double pass one.
  • the overall speedup amounts to about a factor of TWO compared to the previous release F-TexTools-1.0pre2!
  • In all cases the tool nvimgdiff from the new nvidia-texture tools was
    instrumental to assert that the generated images were /identical/ to the original ones after implementation of the speed-up code.
  • overhauled considerably the output logging during operation of the tools.

Let me know
  • how it goes...
  • about your individual speed increase, please!


Throughout, I want to cordially acknowledge active help and good ideas by

Da Woon Jung <dirkpitt2050@users.sf.net> aka dirkpitt aka DW

Enjoy,
Fridger

_________________
Image


Last edited by t00fri on Wed, 24-10-07, 19:37 GMT, edited 3 times in total.

Top
 Profile  
 
 Post subject:
PostPosted: Wed, 24-10-07, 18:56 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
Hi all,

I am pleased to announce the fourth update of the F-TexTools, for making the greatest monster textures for Celestia, directly from the scientific imaging data:

F-TexTools-1.0pre4

along with a binary installer version for Windows users:

F-TexTools-1.0pre4.bin.exe

which --as usual-- can be just installed by double-clicking the archive. Again, a registry entry is made such that the tools can be executed from every directory by just calling their names from the console. You may have to reboot after installation for this to work. Also don't forget to deinstall the previous version (F-TexTools-1.0pre3.bin.exe)!

The first, above archive contains besides the sources, Makefiles and VS-2003.net project files all the binaries for Windows, MAC-OSX-Universal (PPC and Intel) and Linux!

The one-and-only change in this pre4 version is a new routine, png2bin that may be very useful!

With it’s help, any .png textures can be converted to bpp x 8 bit integer binary format bpp=1,3,4), that is
supported by all other F-TexTools.

Here is a little motivating example:

Some of you may still own Mario's@Space-Graphics.com great 32k Mars texture in lossless PNG format: m46_color_32k.png. With the help of my new png2bin routine you can generate in a bit more than only 3 minutes 512 highest quality 1k x 1k, polar optimized VT tiles of level 4 from that texture with this simple commandline:
Code:
> png2bin < m46_color_32k.png|txtiles 3 32768 4

So here is the practical test output, including timing info:
Code:
> time bash -c 'png2bin < m46_color_32k.png|txtiles 3 32768 4'

[png2bin]: Converting PNG format to binary output
           bpp: 3  size:  32768 x 16384

[txtiles]: Input file is a 3x8 bit RGB color map: 32768 x 16384

           Generating  512 optimized VT tiles for level 4
           in PNG format,of size from 256 x 1024 to 1024 x 1024

           PNG_compression level = 1 (range: 0..9)

bin [  1024 rows of 16384 ->   1.38 s]
tile[    32 VT's of   512 ->   2.31 s]
bin [  2048 rows of 16384 ->   3.88 s]
tile[    64 VT's of   512 ->   7.32 s]
bin [  3072 rows of 16384 ->   6.55 s]
tile[    96 VT's of   512 ->  16.82 s]
bin [  4096 rows of 16384 ->   9.27 s]
tile[   128 VT's of   512 ->  26.51 s]
bin [  5120 rows of 16384 ->  12.05 s]
tile[   160 VT's of   512 ->  36.48 s]
bin [  6144 rows of 16384 ->  14.83 s]
tile[   192 VT's of   512 ->  46.56 s]
bin [  7168 rows of 16384 ->  17.63 s]
tile[   224 VT's of   512 ->  56.67 s]
bin [  8192 rows of 16384 ->  20.40 s]
tile[   256 VT's of   512 ->  66.86 s]
bin [  9216 rows of 16384 ->  23.23 s]
tile[   288 VT's of   512 ->  77.15 s]
bin [ 10240 rows of 16384 ->  26.04 s]
tile[   320 VT's of   512 ->  87.35 s]
bin [ 11264 rows of 16384 ->  28.86 s]
tile[   352 VT's of   512 ->  97.20 s]
bin [ 12288 rows of 16384 ->  31.63 s]
tile[   384 VT's of   512 -> 106.95 s]
bin [ 13312 rows of 16384 ->  34.37 s]
tile[   416 VT's of   512 -> 116.57 s]
bin [ 14336 rows of 16384 ->  37.09 s]
tile[   448 VT's of   512 -> 125.81 s]
bin [ 15360 rows of 16384 ->  39.65 s]
tile[   480 VT's of   512 -> 130.84 s]
bin [ 16384 rows of 16384 ->  41.10 s]
tile[   512 VT's of   512 -> 133.05 s]
bash -c 'png2bin < ../m46_color_32k.png|txtiles 3 32768 4'  168.69s user 5.49s system 88% cpu 3:17.02 total

Here are two nice tiles of the 512:

Click for a larger image (1024x1024):
Image
Click for a larger image (1024x1024):
Image

My new commandline tool png2bin is strictly the inverse of bin2png. Here I check with my recent 16k Enceladus texture by first converting PNG -> bin and back bin -> PNG
Code:
> png2bin < enceladus16k.png|bin2png 3 16384 > out16k.png

[png2bin]: Converting PNG format to binary output
           bpp: 3  size:  16384 x  8192
[bin2png]: Converting binary to PNG format output
           bpp: 3  size:  16384 x  8192
bin [  1024 rows of  8192 ->   0.77 s]
png [  1024 rows of  8192 ->  12.37 s]
bin [  2048 rows of  8192 ->   1.54 s]
png [  2048 rows of  8192 ->  25.11 s]
bin [  3072 rows of  8192 ->   2.40 s]
png [  3072 rows of  8192 ->  37.45 s]
bin [  4096 rows of  8192 ->   3.29 s]
png [  4096 rows of  8192 ->  49.49 s]
bin [  5120 rows of  8192 ->   4.12 s]
png [  5120 rows of  8192 ->  61.61 s]
bin [  6144 rows of  8192 ->   4.98 s]
png [  6144 rows of  8192 ->  73.87 s]
bin [  7168 rows of  8192 ->   5.87 s]
png [  7168 rows of  8192 ->  84.72 s]
bin [  8192 rows of  8192 ->   6.72 s]
png [  8192 rows of  8192 ->  94.18 s]

The crucial check on the identity of the two PNG images is then done with the VERY useful tool from the new NVIDIA texture tools package:

> nvimgdiff enceladus16k.png out16k.png

Image size compared: 16384x8192
Total pixels: 134217728
Color:
Mean absolute error: 0.000000 <====Great
Max absolute error: 0.000000
Root mean squared error: 0.000000
Peak signal to noise ratio in dB: 999.000000

Let me know how it goes...

Throughout, I want to cordially acknowledge active help by

Da Woon Jung <dirkpitt2050@users.sf.net> aka dirkpitt aka DW

Enjoy,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Wed, 24-10-07, 19:41 GMT 
Offline
Site Admin
User avatar

Joined: Thu, 30-08-07, 21:25 GMT
Posts: 265
Location: Paris, France
Fridger,

I must confess that I am impressed. Could you perhaps transform my very famous portrait into .bin format?

MonaLisa


Top
 Profile  
 
 Post subject:
PostPosted: Wed, 24-10-07, 22:00 GMT 
Offline
Site Admin
User avatar

Joined: Thu, 30-08-07, 22:52 GMT
Posts: 2727
Location: France, South, not far from Montpellier
Better each time... coool! :o


Top
 Profile  
 
 Post subject:
PostPosted: Wed, 24-10-07, 23:42 GMT 
Offline
User avatar

Joined: Sun, 02-09-07, 16:06 GMT
Posts: 346
Location: Rome, Italy
Well, actually I have only Mario's 16k mars png texture, but nevertheless the results obtained with the new Fridger's tool are outstanding. Give a look at polar caps here:

Image

and here:

Image

No more ugly pinching, full resolution at the poles... wow!!! :shock:
Thanks a lot Fridger, not we have only to wait for good higher res Mars textures. :wink:
Bye

Andrea :D

_________________
Andrea
WorldPoss Sn Team Member
Core 2 Quad Q6600 G0 3.6 GHz- 4 GB DDR2- DELL 2709W 1920x1200-WinXP 32 SP3- VISTA 64- ASUS P5K-E- 8800 GTX 768MB- SATA II 320-300 GB-IDE 1 TB- 169.21- Celestia1.4.1_patch3- 1.5.1-LUA Edu Tools


Top
 Profile  
 
 Post subject:
PostPosted: Wed, 24-10-07, 23:50 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
Andrea,

I thought you would like that new "gadget" ;-)

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Wed, 24-10-07, 23:57 GMT 
Offline
User avatar

Joined: Sun, 02-09-07, 16:06 GMT
Posts: 346
Location: Rome, Italy
t00fri wrote:
Andrea, I thought you would like that new "gadget" ;-)
Bye Fridger

Oh yes, VERY appreciated, thanks a lot.
Bye

Andrea :D

_________________
Andrea
WorldPoss Sn Team Member
Core 2 Quad Q6600 G0 3.6 GHz- 4 GB DDR2- DELL 2709W 1920x1200-WinXP 32 SP3- VISTA 64- ASUS P5K-E- 8800 GTX 768MB- SATA II 320-300 GB-IDE 1 TB- 169.21- Celestia1.4.1_patch3- 1.5.1-LUA Edu Tools


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 29-10-07, 23:29 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 8:49 GMT
Posts: 121
Location: Oxford UK
Well... it all looks very impressive I must say... I've just downloaded the F-Textools and I'm left scratching my head somewhat.

I'm not a newbie when it comes to this computing lark but I just don't know how to get started, or finish, or the bits inbetween.

Yes... I've read the readme!

Sorry.


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 29-10-07, 23:39 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
bh wrote:
Well... it all looks very impressive I must say... I've just downloaded the F-Textools and I'm left scratching my head somewhat.

I'm not a newbie when it comes to this computing lark but I just don't know how to get started, or finish, or the bits inbetween.

Yes... I've read the readme!

Sorry.


Why only "somewhat"? ;-) More is better, hi hi.

bh,

what can I do to get you going? You own a MAC, right. MACs have a perfect UNIX on board and few MACers know how to use it...

Let me know where you got stuck.

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 29-10-07, 23:51 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 8:49 GMT
Posts: 121
Location: Oxford UK
Hehe... thanks for that Fridger... I'll have another go tomorrow, got to get some kip.

Need to download one of the 2gb textures first... can you recommend any particular one?


Top
 Profile  
 
 Post subject:
PostPosted: Tue, 30-10-07, 0:04 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4613
Location: Hamburg, Germany
bh wrote:
Hehe... thanks for that Fridger... I'll have another go tomorrow, got to get some kip.

Need to download one of the 2gb textures first... can you recommend any particular one?


bh,

if you want to do the full tile set of the BMNG Earth, you need the 3 textures I quoted in

http://forum.celestialmatters.org/viewt ... 46&start=1

or here

http://forum.celestialmatters.org/viewt ... sc&start=2

where you may select the month among the BMNG base textures.

For practicing the

5400x2700 'dwarf'-testfiles

http://forum.celestialmatters.org/viewt ... c&start=11

are invaluable!

The required commands for a full set are to be found here

http://forum.celestialmatters.org/viewt ... 46&start=1

(note they refer to the June 2004 base texture. If you prefer another month, you MUST adapt the name!

The latest software packages for download, you find here
http://forum.celestialmatters.org/viewtopic.php?t=70

Good luck, tomorrow ;-)

Bye Fridger

_________________
Image


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 33 posts ]  Go to page 1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group