It is currently Fri, 15-12-17, 12:10 GMT

All times are UTC




Post new topic This topic is locked, you cannot edit posts or make further replies.  [ 1 post ] 
Author Message
PostPosted: Sat, 01-09-07, 15:01 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4514
Location: Hamburg, Germany
Fridgers Column
Welcome Celestians, let me give you a short preview about some exciting software project, that is now shortly before its first public release ...

The following introductory post is a bit historical already ;-) , yet at some important places I have updated the text appropriately. It's essentially the first preview of my new texture tools from Fridger's Column in our Celestial Matters Website, along with some people's comments and my answers.

I hope this will make a good start for the vivid and updated discussions to come!

Preview of a new Toolset for Processing Monster RGB(A) textures (BMNG,...)

Besides my (successful) nmtools package, a corresponding toolset is clearly needed that can efficiently process the huge RGB texture sets of Earth (BMNG), Mars, etc for use in Celestia. These raw data are professionally published as (gzip-packed) 3x8bit binary raw formatted files in "interleaved" RGB ordering. The latter means that the R,G,B colour values are encoded this way:

interleaved: [font=Courier New]R1 G1 B1 R2 G2 B2 R3 G3 B3 ...[/font]

instead of:

planar: [font=Courier New]R1 R2 R3 ...G1 G2 G3 ... B1 B2 B3...[/font]

Note smaller textures of this format can also be loaded and written in Photoshop via the so-called .raw format! But Photoshop can hardly handle e.g. 85k BMNG base textures of size 3x86400x43200! That's what we are after, however, aren't we. Such monsters have unpacked a size of 11 GB ! The gzip packed, published files are "only" ~ 2.2 GB in size.

So I have set out to code a set of versatile command-line tools that can handle all the needed tasks of processing such monster textures FAST and in HIGHEST quality with normal home computers. Very much the same philosophy as in case of my nmtools. The new tools will produce VT's with ALL optimizations included in one single custom step. They also allow you to just reduce the 85k texture to the nearest power-of-2 sized texture (64k) and then to produce textures that are smaller by successive factors of 2...(32k, 16k, 8k, 4k, 2k).

The greatest thing about my new tools is that they are again VERY fast and versatile via their pipe mechanism. For example, reducing the gzipped 85k monster BMNG "flat" base texture (June 2004) - world.200406.3x86400x43200.bin.gz - from The Nasa Arctic Server to the nearest 64k and then successively to 16k just involves the following simple piped command

[font=Courier New]gzip -dc < world.200406.3x86400x43200.bin.gz|tx2pow2 3 86400|tx2half 3 65536|tx2half 3 32768 > world.200406.3x16384x8192.bin [/font]

The first argument in 'tx2pow2' and in 'tx2half' denotes the number of color channels per pixel. It can be 1 (grayscale), 3(RGB) or even 4 (RGB + alpha =RGBA). The second argument denotes the texture width in pixels.

This was just one simple example application. There are quite a few more tools, including of course one for making optimized VT tiles ('txtiles') and one that mounts specular grayscale textures as the alpha channel of the RGB base texture ('tx2rgba'). So all tiles can be made in 4-channel RGBA format. This boosts performance and saves GPU memory!

The whole above piped action chain takes only ~17 minutes on my machine!
Here is a partial view from the resulting 16k BMNG texture in exquisite quality:

Image (click for full-size)

Obviously, you now may easily include ALL 12 seasonal BMNG textures in Celestia in highest quality (provided your harddisk is large enough ;-) )!

In this short preview, I wanted to let you know that I am almost done with this project! As with the nmtools, the plan is to make the new tools available for download in Celestial Matters and in this Forum, along with a user manual, and to support ALL operating systems.

After you all followed my nmtools tutorial (and succeeded! ;-)), working with the new tools will be VERY easy for you.


Discussion on this from my old blog:

-----------------------------------
8/23/2007, 22:27
Where do I put the Files in Celestia?
Posted by: Alex

Hello all, I followed the instructions and used nmtools and everything went okay.Got the files in level0 to level5 directories. Now what? I can't find help on how to use those files in celestia(I have 1.4.1) and the author talks about celestia 1.5.1(Can't find it for download).Anyway my idea was to use these files with celestia and maybe another program to have the entire world in Hi-res on my laptop. Any ideas welcome Thank you very much

-----------------------------------
4/19/2007, 05:46
dxt5nm vs. png sizes
Posted by: Tech2000

Ok Fridger, can you help me get this straight than.

I use your nmtools in a shellscript of mine, there seems to be something strange going on with the net at the moment so I can't post it for you, but it's very basic and works perfect.

The strange thing is that the finished PNG (904 969 477 bytes) is smaller than the dxt5nm (1 480 691 760 bytes) one. I thought that the dxt5nm version should be more size economic than the png version.

Is this normal and I still haven't got it the hang of dxt5nm or is something fishy going on here?

nmstiles 4 6378.140 32768 2.5 < world32k.bin
nvdxt -quality_highest -file *.ppm -nomipmap -dxt5nm
ren *.dds *.dxt5nm

Cheers, Anders

-----------------------------------
4/16/2007, 23:44
Normalmap file format
Posted by: Fridger

Anders,

traditionally, the lossless PNG gives the /best quality/ for normalmap VTs. It's still reasonably fast for VTs. However, the recent DXT5nm format is almost as good, yet shares all the advantages of hardware supported DXT compression...

I have explained several times in the forum why DXT5nm is so good for normalmaps.

Bye Fridger

-----------------------------------
4/16/2007, 20:25
Posted by: Tech2000

Which file format is best for normal maps and surface textures in respect of quality, performance & file size?

Bye, Anders

-----------------------------------
4/15/2007, 21:37
specular channel
Posted by: Fridger

Yes, Anders,

In my new txtools, the rgb and alpha channel VT tiles are automatically generated and then they are mounted together into RGBA tiles. This is in many respects the best and most economic option.

Cheers, Fridger

-----------------------------------
4/14/2007, 17:44
Specular map in alpha channel.
Posted by: Tech2000

Hi Fridger! I have a question for you. I found this info at the Celestia Motherlode: "Specular Maps: Specular maps mark surface areas covered by water, so Celestia can render these ares with specular reflections, i.e. they appear more shiny. This helps tremendously for rendering realistic surfaces. Note that a few surface texture already contain the specular map (as an alpha-layer)." Does your txtools have the posibility to integrate specular maps in the alpha channel?

Best regards,
Anders Pamdal

-----------------------------------
3/8/2007, 01:43
Posted by: t00fri

Fightspit,

Oh sorry, I overlooked that you were interested in DXT5 vs. DXT3.

DXT3 and DXT5 both have a compression ratio of 4:1 (while DXT1c/DXT1a) has 8:1.The algorithms mainly differ in alpha channel compression. DXT3 is often called "RGB texture with explicit alpha" while DXT5 is "RGB with interpolated alpha". The RGB part is the same for DXT1,3 and 5. Hence if you only want to compress RGB, use DXT1 because of the 8:1 compression with the rest being identical.

Bye Fridger

-----------------------------------
3/7/2007, 22:51
Posted by: Fightspit

I don't talk about the DXT5nm but DXT5 only (no normal map), what it is the difference with DXT3 ?

Fightspit

-----------------------------------
3/7/2007, 21:09
DXT5nm
Posted by: t00fri

Fightspit,

DXT5nm is only for compressing normalmaps. It exploits special properties of normalmaps, like the texel normalization condition r*r + g*g + b*b = 1

I use DXT3 compression throughout for the RGBA composite of base texture and spectexture tiles.

Bye Fridger

-----------------------------------
3/7/2007, 18:21
Posted by: FIGHTSPIT

I want to know if it will be better to have the tiles in DXT5 instead of DXT3 ?

Fightspit.

-----------------------------------
2/27/2007, 23:58
Posted by: Guest Jo

After Pierre the author of xnview this is the correct syntax: nms 6378.140 4096 20.0 < srtm_ramp2.world4k.bin | nconvert -out png stdin For me this does not work. I sent him example ppm-files, but I don´t know if something comes out of that. Think I am not the expert to discuss this with Pierre... See: http://newsgroup.xnview.com/viewtopic.p ... 289bfd483e

-----------------------------------
2/25/2007, 19:31
Posted by: Guest Jo

Ok, thank you for confirmation, I posted to xnview forum, lets see what happens. Many thanks again for the hard work you invest into celestia community...

-----------------------------------
2/24/2007, 19:24
piping
Posted by: t00fri

Hi Joerg,

good you bring this up. I have never been able to locate an option for nconvert that switches to STDIN!

Perhaps it's worth asking the author of xnview/nconvert?

Anyhow, I have meanwhile completed the coding of a whole new suite of tools => txtools, along with shell-scripts for DOS and Linux that allow doing all textures in one go. In particular I coded a DevIL-based tool that does directly DXT5nm encoding and also automatically packs the spec texture into the alpha channel of the rgb base texture and subsequently compresses the resulting RGBA tiles as DXT3. All this is very convenient, fast and produces VERY good quality.

Unfortunately there is still some stagnation as to MAC-OS, since DevIL is not yet available as a binary release for MACS (despite the promises of the DevIL people). Things work very nicely already for Linux AND Windows, for which I shall again release an installer-based version.

Moreover, the compile-farm feature of SourceForge has been closed down very recently. So I cannot compile anymore my tools for MAC's on their machines...

Bye Fridger

-----------------------------------
2/24/2007, 15:30
piping
Posted by: Guest Jo

Ok, reading the help-files more carefully I saw that nms has stdout but nmstiles doest.t have. So I tried: nms 6378.140 4096 20.0 < srtm_ramp2.world.4096x2048.bin | nconvert -out png stdin but it doesn´t work. From the xnview-forum I read that stdin stdout doesn´t work for all formats. Can anyone confirm that ? (BTW This is no "problem". I just wan´t to know cause I´m always asking when I don´t know..)

-----------------------------------
2/22/2007, 23:20
piping
Posted by: Guest Jo

I have followed the tutorial and all works very well. Now I wrote a little batch file to automatize the whole process. Is it possible to do the png or dxt5nm conversion in a pipe with the nmstiles instead of using a separate command ? What syntax will I need for | nconvert... or | nvdxt... nconvert -help didn´t help me.

Best wishes Jörg

-----------------------------------
2/16/2007, 19:35
Startygers question
Posted by: t00fri

Startyger,

sorry but I hardly understand what you are alluding at.

I am concerned with my tools with a job that cannot be better than the original input imaging. My tools serve to prepare the latter for use in Celestia.

What do you mean with "core images"?

Bye Fridger

-----------------------------------
2/16/2007, 12:06
Posted by: t00fri

Ozark,

congratulations for successfully finishing the tiles. Since there are a number of standard versions how to implement the VT tiles, I strongly urge you to read the excellent docs about this. It's really easy, but hard without ever looking into the available documentation about Celestia!

Of course I have to assume that the users know what to do with the resulting VTs ;-).

I guess Selden has the best instructions besides the official manual.

http://www.lns.cornell.edu/~seb/celestia/

Soon I will publish further exciting tools that allow to generate the FULL set of >= 64k VTs (base, spec, normalmap) with all optimizations /automatically/ (batch script), fast and in highest quality.

Bye Fridger

-----------------------------------
2/14/2007, 14:49
very much a newbie.. but...
Posted by: Startyger

I was wondering... since apps like chocoflop (chocoflop.org) can work on 128bit (HDR?) images via the macOS's quartz engine... would it be possible... to make an image unit that can do the same thing via core image? this might be a stupid question... but... if it's possible... wouldn't that be nice?

-----------------------------------
2/13/2007, 20:30
successful use of nmtools but now what?
Posted by: Ozark

Hey Fridger, I must admit I am quite new to all of this, and I have virtually no skill programing anything except my VCR. That said thank you very much for the tutorial on reducing that behemoth of an image file, everything worked just fine for me, except I have a directory full of a chopped up image that i don't know how to put to use. I have checked the forums at http://www.motherlode.net but haven't found he help i need. Perhaps I missed it, but do i put the files in a .ssc file or what. Please help a point in the right direction would be most appreciated.

Ozark

-----------------------------------
1/30/2007, 11:05
nmtool
Posted by: Fightspit

Try it : http://www.celestialmatters.org/cm/host ... -1.0.1.zip

-----------------------------------
1/30/2007, 03:59
nmtools
Posted by: TheOWHSM

Hi, Fridger. Sorry, but I can't download your nmtools. I already downloaded 85k height map, but not yet downloaded nmtools. So, send nmtools to my e-mail hisnook@mail.ru, or show other link.

-----------------------------------
1/24/2007, 20:03
Posted by: RRA

That's what I thought , in the mean time I managed to establish an RSS feed, seems to work OK, If you need help testing the code for Windows I am glad to help.

René

-----------------------------------
1/24/2007, 19:54
new toolset
Posted by: t00fri

René,

no, as the name indicates (nm = normalmap), the nmtools package specializes on processing large normalmaps. For processing RGB textures I am going to publish a different package, yet with tools of analogous functionalities. I still have to find a good name ;-)
Meanwhile, I have completed the coding of the new tools. The next steps are compiling and testing for Windows and Linux, which I do myself. The OSX binaries have to be again compiled by Dirkpitt, since I do not have a Mac.

Then I will have to write a user manual here in CM. When the new tools will actually appear depends on my spare time. But I am optimistic that it won't be very long. Just stay tuned. How about establishing a RSS feed and you will not miss anything going on here...

Bye Fridger

-----------------------------------
1/24/2007, 08:29
toolset
Posted by: Guest CM

Hello Fridger, the toolset you are talking about: "a corresponding toolset is clearly needed" I guess isn't included in the nmtools toolset (version 1.01) or is it ? I can't find the executable you describe , for instance res2pow2_rgb and so on. Where and when will it be available ??

René Ramekers

-----------------------------------
1/23/2007, 00:03
Posted by: Fridger

Hi "Celestial" friends,

after quite a bit of further writing, overhauling and testing for several operating systems, I am happy to announce that my detailed tutorial about the use of the 'nmtools' is now essentially completed!

Also, please find a slight update of my nmtools to version 1.0.1 for download!

The change only affects Windows users when installing nmtools by means of the Setup.exe installer. The installer script now adds an entry into the Registry which makes the nmtools known as executables in every application directory under console operation! This is very handy.

At this point, there is plenty of "playground" for you to practice and/or to attempt making your own highest quality 64k normalmap in form of VT tiles according to my step-by-step instructions.

I also devoted some efforts to discuss how to convert to the new compressed DXT5nm format that is supported in Celestia (pre x) 1.5.0. There are many advantages of this format, while the image quality remains still VERY high.

"Fridger's column" --for now-- is mainly devoted to a discussion of the nmtools along with questions some of you may have! In a little while I shall address various other "hot topics" as well ...

Enjoy & good luck with your 'nmtools' experiments

Cheers, Fridger

-----------------------------------
1/15/2007, 23:45
Activating the register for gzip
Posted by: Guest CM

I follow the path until the Control Panel and in Advanced,NmTools is in Register to be activated,but Gzip not. I simply installed Gzip to the C partition,changing for the D partition,but it didn´t work. There was a right path for Gzip,but after I installed in D partition;the path is no longer there.What can I do?

-----------------------------------
1/4/2007, 02:43
RE:nms- tools
Posted by: John Van Vliet

i keep fergetting to use my name , too used to a cookie doing it the bug report is mine

-----------------------------------
1/2/2007, 23:01
nms- tools
Posted by: Guest CM

hi Fridger seeing as image magick is still broked as to converting 16 bit unsined to 16 bit sined and msb to lsb i did a very quick hach to nms ( nmsu) for unsined it may be my hack but i noticed that for a 4k map there is a 3 pixle ofcet to the right and for a 8k map there is a 32 pixle offcet to the right -------------------------------------------------- float readS16(FILE *in) { long b2; fread(&b2, 2, 1, in); if (byteSwap == 1) b2 = (32767-((b2 & 0xff00) >> 8)) | (32767-((b2 & 0x00ff) << 8)); return (long) b2; -----------------------------------------

-----------------------------------
12/24/2006, 11:10
About an appropriate base texture
Posted by: FIGHTSPIT

I have tested your nmtool and it is very impressive ! There is someone called 'RVS' (in shatters forum) who recentely provides me a tool to generate virtual base texture (64k) with optimisation and I think it is possible for you to do same tool for people who want to use your great 64k normal map. Thank for you beautiful work!

Fightspit

-----------------------------------

_________________
Image


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic This topic is locked, you cannot edit posts or make further replies.  [ 1 post ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group