It is currently Wed, 22-11-17, 9:17 GMT

All times are UTC




Post new topic Reply to topic  [ 40 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: Scripting
PostPosted: Tue, 04-09-07, 18:34 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
The use of the Texture Tools can be greatly simplified by means of scripts, such that whole texture sets are fabricated with just a few mouse clicks or commands. These scripting tasks can be accomplished with various languages like UNIX shell scripting, DOS commands, PERL etc.

Looking forward to your creative participation!

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 21-09-07, 11:28 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
Hi all,

let me write up typically all the commands that would be necessary for generating a full-sized (64k) set of VT tiles for the base & spec textures. For testing purposes, it might also be useful to transscribe the file sizes given below to the "dwarf" 5400x2700 data set I prepared for you here:

http://www.forum.celestialmatters.org/v ... 4&start=11

Well, with a few scripting commands the whole procedure that I am simply listing below can be entirely automatized. I did not yet add such scripts to the archive to stimulate some creativity from the user side ;-) .

So here are the obvious commandline steps:
(in Unix scripting language the whole procedure below can be done with just a few lines!)

To economize on the used harddisk space I assume that the familiar 'gzip' command-line utility is installed on your machine. It is installed by default in Linux and MAC's. For Windows you find it here:
http://gnuwin32.sourceforge.net/packages/gzip.htm

-----------------------------------------------------------------
Throughout, I assume that on the console, you work in the SAME directory where the three INPUT 85k monster textures from the snowy. arctic NASA server,
------------------------------------------------------------------

http://mirrors.arsc.edu/nasa/world_big/ ... 200.bin.gz
http://mirrors.arsc.edu/nasa/landmask/w ... 200.bin.gz
http://mirrors.arsc.edu/nasa/topography ... 200.bin.gz

are located.

First of all, you must create a set of tile directories with

Code:
mkdir level0
mkdir level1
mkdir level2
mkdir level3
mkdir level4
mkdir level5


Next turn to the 85k input texture files:

You MUST first unzip the watermask file via
Code:
gzip -d world.watermask.86400x43200.bin.gz


Then you create a much improved, binary specmap.bin graytexture:

Code:
gzip -dc < srtm_ramp2.world.86400x43200.bin.gz | specmap 86400 world.watermask.86400x43200.bin 0.12 1 > specmap.bin


NOTE: Watch out to do the correct byteswapping here, depending on your machine (last parameter in the specmap tool!!

Normal Windows & Linux PC's & MAC Intels : NO CHANGES in the above command line!
(Older) Power PC MACS: delete the last parameter ('1') of specmap in the above command line.


Next you mount the specmap.bin file as alpha channel on top of the RGB base texture in a 64k binary RGBA output texture:

Code:
gzip -dc < world.200406.3x86400x43200.bin.gz | tx2rgba 86400 specmap.bin | tx2pow2 4 86400 > level5/world.200406.3x65536x32768.RGBA.bin


NOTE: The directory slashes are for Unix (Linux, MAC) consoles here. For Windows/DOS just replace with level5\world.200406.3x65536x32768.RGBA.bin and so on...

scale the 64k RGBA texture to half size:

Code:
tx2half 4 65536 < level5/world.200406.3x65536x32768.RGBA.bin > level4/world.200406.3x32768x16384.RGBA.bin


NOTE: The directory slashes are in Unix notation (Linux, MAC) here. For Windows/DOS just replace with level5\world.200406.3x65536x32768.RGBA.bin and level4\world.200406.3x32768x16384.RGBA.bin
and so on...

Windows: Just replace ALL '/' directory slashes by '\' below! E.g. also in 'cd ../level4' etc.


scale the 32k RGBA texture to half size:

Code:
tx2half 4 32768 < level4/world.200406.3x32768x16384.RGBA.bin > level3/world.200406.3x16384x8192.RGBA.bin



scale the 16k RGBA texture to half size:

Code:
tx2half 4 16384 < level3/world.200406.3x16384x8192.RGBA.bin   > level2/world.200406.3x8192x4096.RGBA.bin


scale the 8k RGBA texture to half size:

Code:
tx2half 4   8192 < level2/world.200406.3x8192x4096.RGBA.bin    > level1/world.200406.3x4096x2048.RGBA.bin


scale the 4k RGBA texture to half size:

Code:
tx2half 4   4096 < level1/world.200406.3x4096x2048.RGBA.bin    > level0/world.200406.3x2048x1024.RGBA.bin


Then you do the tiles. If you want to stay with PNG finally, you can increase the PNG compression factor if you are short of space. Otherwise one will want to use the new cross-platform nvcompress utility from NVIDIA later, to do high-quality DXT5nm/DXT3 compression of the tiles..


First generate

2048 tiles in level5:

Code:
cd level5
txtiles 4 65536  5  2.7 < world.200406.3x65536x32768.RGBA.bin   


512 tiles in level4:

Code:
cd ../level4
txtiles 4 32768  4 2.7 < world.200406.3x32768x16384.RGBA.bin   


128 tiles in level3:

Code:
cd ../level3
txtiles 4 16384  3 2.7 < world.200406.3x16384x8192.RGBA.bin   


32 tiles in level2:

Code:
cd ../level2
txtiles 4   8192  2 2.7 < world.200406.3x8192x4096.RGBA.bin   


8 tiles in level1:

Code:
cd ../level1
txtiles 4   4096  1 2.7 < world.200406.3x4096x2048.RGBA.bin   


2 tiles in level0:

Code:
cd ../level0
txtiles 4   2048  0 2.7 < world.200406.3x2048x1024.RGBA.bin   


Here you are done ;-) .

Note that the resulting tiles have the specular texture as alpha channel of the RGB base texture tiles. This is VERY favorable for speed and storage!
The lower levels go VERY fast indeed...

Good luck,
Fridger


Last edited by t00fri on Tue, 06-05-08, 14:12 GMT, edited 2 times in total.

Top
 Profile  
 
 Post subject:
PostPosted: Sun, 23-09-07, 12:29 GMT 
Offline
User avatar

Joined: Sun, 02-09-07, 16:06 GMT
Posts: 346
Location: Rome, Italy
t00fri wrote:
Hi all,let me write up typically all the commands that would be necessary for generating a full-sized (64k) set of VT tiles for the base & spec textures.

Quote:
512 tiles in level4:

cd ../level4
txtiles 4 32768 4 2.7 < world.200406.3x65536x32768.RGBA.bin

Fridger
:wink:

Fridger, working with your fantastic F-TexTools-1.0pre2 (with modified new tx2rgba.exe :wink: ), I found a litthe typo in your input commands for level4 tiling: the right one should be

512 tiles in level4:

cd ../level4
txtiles 4 32768 4 2.7 < world.200406.3x32768x16384.RGBA.bin



As you see I learned your lesson at nmtools time, and now I have a full set of VT tiles with an astonishing quality.
THANK YOU VERY, VERY, VERY MUCH! :D
Bye

Andrea :D

_________________
Andrea
WorldPoss Sn Team Member
Core 2 Quad Q6600 G0 3.6 GHz- 4 GB DDR2- DELL 2709W 1920x1200-WinXP 32 SP3- VISTA 64- ASUS P5K-E- 8800 GTX 768MB- SATA II 320-300 GB-IDE 1 TB- 169.21- Celestia1.4.1_patch3- 1.5.1-LUA Edu Tools


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 23-09-07, 12:46 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
Hi all,

please note, the typo found by Andrea in one of my commands

WRONG wrote:
cd ../level4
txtiles 4 32768 4 2.7 < world.200406.3x65536x32768.RGBA.bin


has been corrected above.

RIGHT wrote:
cd ../level4
txtiles 4 32768 4 2.7 < world.200406.3x32768x16384.RGBA.bin


So now it should all be working fine, since Andrea made it successfully through these commands!

Cheers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 7:23 GMT 
Offline
User avatar

Joined: Sun, 21-10-07, 5:43 GMT
Posts: 21
Location: Minnesota, USA
Hi Fridger,

I have created a automated script for the TEST FILES. :D :D
This is my first script EVER so the coding is quite "crude". However it will create (Rgba) VT's in dds (dxt1) format, and the Srtm VT's in dds (dxt5nm) format with options to change to png or any other type of Image file one wishes to use. And puts them in thier respective folders and subfolders for an "extras" addon as long as the RGB file, Srtm file , and Watermask file are in the same Folder as My .bat file. Oh yeah it also records times for each tex to complete, though i dont know how accurate the results of them are.. they seemed kinda fishy to me.
The whole process takes les than 2 minutes

I know you don't want anyone to upload the complete texture packs to the motherlode, so I thought I would ask you if you had any reservations with me making this file available here, considering it's pretty much one step away from uploading a texture pack. :wink:

This script was made with your F Tex tools 1.0pre3 package, and nmtools 1.5pre1 package.

bye,

Ozark

_________________
Celestia 1.6 4GHZ E6550
4GB DDR3 ram
Windows XP
512 GeForce GTS 250 - NVIDIA


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 9:28 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
Ozark wrote:
Hi Fridger,

I have created a automated script for the TEST FILES. :D :D
This is my first script EVER so the coding is quite "crude". However it will create (Rgba) VT's in dds (dxt1) format, and the Srtm VT's in dds (dxt5nm) format with options to change to png or any other type of Image file one wishes to use. And puts them in thier respective folders and subfolders for an "extras" addon as long as the RGB file, Srtm file , and Watermask file are in the same Folder as My .bat file. Oh yeah it also records times for each tex to complete, though i dont know how accurate the results of them are.. they seemed kinda fishy to me.
The whole process takes les than 2 minutes

I know you don't want anyone to upload the complete texture packs to the motherlode, so I thought I would ask you if you had any reservations with me making this file available here, considering it's pretty much one step away from uploading a texture pack. :wink:

This script was made with your F Tex tools 1.0pre3 package, and nmtools 1.5pre1 package.

bye,

Ozark


Many thanks, Ozark,

of course, everyone is encouraged to upload their scripts! So others can possibly improve them and use them.

Clearly I don't want the textures to be uploaded to the Motherlode, because

  • Doing them yourself means always starting from the ORIGINAL published imaging data, which corresponds to the proper scientific documentation spirit of Celestia
  • Assures always equal high quality and allows fast later improvements, all based on the ORIGINAL data
  • Has a substantial learning effect
  • I am deeply convinced that the Motherlode is the WRONG concept. Since the CM sites are on the same Ibiblio server with virtually unlimited storage, let's see what will grow out of CM in the future ;-)


So this thread is precisely for exchanging know-how about scripting for the tools.

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 11:11 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Here is a dumb question, something I should already know that is. After creating a list of files in a directory, How do I get a program to go through that list one at a time or is that something I need to write into the program?
Code:
ls *.hgt | sed s/.hgt// > files.lis
/home/cartrite/specmap-work-area/works2/srtm3/part1b 1201 1201 < $1.hgt > /home/cartrite/specmap-work-area/works2/srtm3/$1.bin -batchlist=files.lis

Attemp2 I get a syntax error on this one line2
Code:
#!/bin/csh
for i ( *.hgt )
  set base=`basename $i.hgt`
  set new="$base.spec.bin"
  /home/cartrite/specmap-work-area/works2/srtm3/part1b 1201 1201 <  $i.hgt > /home/cartrite/specmap-work-area/works2/srtm3/$new
end
done:
  exit 0

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:03 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
cartrite wrote:
Here is a dumb question, something I should already know that is. After creating a list of files in a directory, How do I get a program to go through that list one at a time or is that something I need to write into the program?
Code:
ls *.hgt | sed s/.hgt// > files.lis
/home/cartrite/specmap-work-area/works2/srtm3/part1b 1201 1201 < \$1.hgt > /home/cartrite/specmap-work-area/works2/srtm3/\$1.bin -batchlist=files.lis

cartrite


cartrite,

you don't even need to create that list. The secret is the 'find' utility that is precisely doing this for you.
So here is what you do under Linux:
Code:
> cd ~cartrite/specmap-work-area/works2/srtm3/
> find . -name "*.hgt" -exec ./doit \{\} \;

The doit script is trivial:
Code:
-------------------doit--------------------------
#! /bin/sh

name=${1%.*} #full path to file specified by $1, minus extension
./part1b 1201 1201 < $1 > ${name}.bin
---------------------------------------------------

You must make it executable, of course with
Code:
> chmod a+x doit

before using it.

Bye Fridger

_________________
Image


Last edited by t00fri on Sat, 27-10-07, 12:20 GMT, edited 1 time in total.

Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:11 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
cartrite,

of course, for experimenting you may enjoy my 'bin2png' utility from the F-TexTools.

By writing another trivial script

Code:
--------------doit_png----------------------
#! /bin/sh

name=${1%.*} #full path to file specified by $1, minus extension
./part1b 1201 1201 < $1|bin2png 1 1201 1201 > ${name}.png
------------------------------------------------


and making it executable, too ;-)

By using 'doit_png' instead of 'doit' in the above 'find' statement, you directly get PNG output for inspection!

That's the beauty of the pipeline mechanism...

Here is an example of what I get with this simple prescription

Image

after inverting the result from part1b and reducing it to 700x700 px, in order not to screw up the display.

Bye Fridger

_________________
Image


Last edited by t00fri on Sat, 27-10-07, 12:20 GMT, edited 2 times in total.

Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:19 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Thanks Fridger,
I'll try that too. This also works too though. My first mistake was that I didn't have csh in the first line. My second mistake was a "hidden character" or cr in the for i ( *.hgt ) line. I was getting a "badly placed "()" error.
Code:
#!/bin/csh
foreach i ( *.hgt )
  set base=`basename $i.hgt`
  set new="$base.spec.bin"
  /home/cartrite/specmap-work-area/works2/srtm3/part1b 1201 1201 <  $i > /home/cartrite/specmap-work-area/works2/srtm3/$new
end
done:
  exit 0

I'm going to try a run on the North America directory from the SRTM3.
I noticed your second response before I posted this. And yes, the pipeline is the way to go. I'm going to refine this before the real run.
Edit:Just seen the image after posting. I was think part 2 may only need to be run once. I found that when scaling down the images to match the 500m data from the BMNG file, it cleans up a lot of the single white pixels.
Thanks Again.
cartrite


Last edited by cartrite on Sat, 27-10-07, 12:23 GMT, edited 1 time in total.

Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:22 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
Did you also see my example PNG output? I had edited it in a little later.

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:25 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Yes, thats looking pretty good. Did you try the desert region that had water in dry areas on the watermask? So you got that from just running part 1 and turned it to png? Parts 2 and 3 just look for single white pixels surrounded by black ones. Difference between them is that part 3 outputs pgm files. There may be areas where they are needed though. Like eastern Canada for one. I got a lot of noise in Canada.
cartrite


Last edited by cartrite on Sat, 27-10-07, 12:36 GMT, edited 1 time in total.

Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:35 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
cartrite wrote:
Yes, thats looking pretty good. Did you try the desert region that had water in dry areas on the watermask? So you got that from just running part 1 and turned it to png?
cartrite


Yes, it is the direct output from doit_png. But who knows whether it's also correct?? That's the crucial issue, though.

I got to find out first, how to find those tiles with the "dessert lakes" ;-)

Cheers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:45 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Looking at that image again, I noticed that it seems to be inverted. Are the white areas lakes or are they islands in the ocean? If the black area is the ocean then it shouldn't it be white?


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 27-10-07, 12:47 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
cartrite wrote:
Looking at that image again, I noticed that it seems to be inverted. Are the white areas lakes or are they islands in the ocean? If the black area is the ocean then it shouldn't it be white?


Yes I inverted it (see my description above). I had no idea what it was and this way it looked better ;-)

Bye Fridger

_________________
Image


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 40 posts ]  Go to page 1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group