It is currently Mon, 21-10-19, 2:36 GMT

All times are UTC




Post new topic Reply to topic  [ 148 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7 ... 10  Next
Author Message
 Post subject:
PostPosted: Mon, 05-11-07, 21:49 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4652
Location: Hamburg, Germany
Amazing! Hang on ;-) .

Chers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Tue, 06-11-07, 13:45 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I'm not sure what the best way to do this is. There are a couple of different ways. I'm leaning towards creating tiles from the 84k spec map created from the zero values in the srtm. A1 through D2 like the Blue Marble tile scheme. I tried to run the test tile on the whole map but it took forever to do one tile. Then the gimp could not open it cause after running gdal_translate the file becomes a geo_tiff. The gimp could open smaller files after complaining but can't open the big one. Too many error messages I guess.

I've noticed a problem with creating a spec map from zero values in the srtm. There are some areas in the continents that have white marks which may mean that there are zero values in the srtm that have nothing to do with water. The newer version of the file, if there is one, may have cleaned this up. If not then running the files on "blank" 21600x21600 files would probably work out better. I'm not sure if this is the best way though because gdal_rasterize may not use the best method to scale the files down. I'll have to check. The source code for gdal_rasterize could be altered to force cubic convolution if it's not already being used.

Since these files only cover from 60 north to 60 south, the watermask file would probably still be needed to cover those areas. Can the specmap program do this? We would have to convert the finished tiles from geo-tiff to raw binary. The raw 84k file is not large when zipped so it would probably be best to create a spec map for all to use and host it here? Then everyone could just run tx2rgba.

The script to do this would have to use the file name without the extension "layername -l" in the command. The attribute can always be "WB". EDIT: I found that the "WB" attribute is not always present in the files. I used the FACC_code instead.
The zip files would also need to be sorted and placed in there own directory depending on which tile (A1 through D2) they belong to.

Anyhow, I'm going to wait till you get back. Hope your having a good time in Mexico.

cartrite


Last edited by cartrite on Thu, 08-11-07, 13:21 GMT, edited 1 time in total.

Top
 Profile  
 
 Post subject:
PostPosted: Thu, 08-11-07, 1:32 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I decided to go ahead with testing this even if there may be a better way or better data in the near future. Here are the commands used and a 4k version of the 21600x21600 a1 tile I did. The run took approximately 5 hours.
I was looking for a way to limit the find command to only find the files I wanted to process but came up with this instead. This script looks at all files in the west directory but only processes the files for tile a1. I wrote a separate script for each tile.
First I ran a program I wrote last year to use 0 values on the 84k srtm heightmap to create white oceans and coastlines. Then I used Image Magik to crop it into 21600x21600 tiles. Then I needed to turn a1 into a geo-tiff file.
Code:
gdal_translate  -a_ullr -180 90 -90 0 oceanspecmap-west-a1.png a1.tif

Then after unzipping the swbd west files I ran this script on a1.tif.
Code:
#! /bin/sh

name=${1%.*} #full path to file specified by $1, minus extension
name1=${1:2:8}
x=${1:3:3}
y=${1:6:1}
if [ ${x} -gt 090 -a "${y}" = "n" ]; then
gdal_rasterize -b 1 -burn 255 -l ${name1} -a FACC_code ${name}.shp a1.tif
echo "${name1}"; fi

Thank's to Fridger, I used this command to run it.
Code:
 find . -name "*.shp" -exec ./batch-a1 \{\} \;

Then I ran this to turn it back to png.
Code:
gdal_translate -of png a1.tif oceanspecmap-a1.png

Thumb of a 4k file. The original is 21600x21600.

Image

EDIT: I ran b1 last night. I'm currently running a2. I should have the west side done by later tonight or tomorrow. The whole map should be done some time this weekend. I'll assemble them and convert the file to raw binary, gzip and upload to my site for now so others can test this.
EDIT: fri nov9,07 Running ahead of schedule. Should be finished later today sometime when I will post a link to the finished bin file. If my site is up and running?
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 09-11-07, 21:03 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I finished processing the swbd files into 8 png files 21600x21600 but I hit a snag though! I can't figure out Image Magik to assemble the tiles into 1 image.
I run this command for the top half and IM creates a RGB image with 48bpp depth. The files are getting too large with that depth and it takes forever.
Code:
montage  -geometry 21600x21600+0+0 oceanspecmap-a1.png -geometry 21600x21600+0+0 oceanspecmap-b1.png -geometry 21600x21600+0+0 oceanspecmap-c1.png -geometry 21600x21600+0+0 oceanspecmap-d1.png -tile 4x1 top.png

I'm currently running this command but it "looks" like it's doing the same thing. Like it's ignoring the -depth option.
Code:
montage  -geometry 21600x21600+0+0 oceanspecmap-a1.png -geometry 21600x21600+0+0 oceanspecmap-b1.png -geometry 21600x21600+0+0 oceanspecmap-c1.png -geometry 21600x21600+0+0 oceanspecmap-d1.png -tile 4x1 -depth 8 top.png

I gave up on Image Magik. I'm now starting a long painful process of doing the mosaic with the gimp. Good Ol Gimp.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 10-11-07, 7:34 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Done. I started a new thread in the Textures of Celestial Bodies section. I'll upload the binary file as soon as my site is up and running again. It has been going down a lot lately.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 11-11-07, 14:22 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
One of the things that concern me is that the full resolution swbd files should have a minimum lake size of 180 meters x 600 meters. That translates to about 6x20 pixels. When gdal_rasterize reduces the resolution from 30 meters per pixel to 500 meters per pixel, a lot of these smallest lakes are being reduced to a single line of pixels or less. This may be interpreted as "noise" by the Celestia user. Like the Northern Canada area for example.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 12-11-07, 2:51 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I'm not sure what could be done to add the missing data from 60 to 90 north.
Here are a couple of tiles showing the mismatch between the 2 datasets.

Image

Image

Image

The best approach would probably be to write some code that adds this data during specmap creation process with the specmap tool. The SWBD data starts on line 7200. Shouldn't be that hard. I don't think I want to physically add this data to the SWBD_Watermask file. These images also show the issue I mentioned in my previous post about single pixel lakes.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 12-11-07, 4:36 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I just thought of something that may work. If I crop 7200 lines from the top of the July BMNG world file and identify pixels values for lakes and rivers in each band, then I could write some code to change those values to 0 and everything else 255. July would be the best month for this since there is the least amount of snow. Think this would work? It won't be 100% but probably will be better then using the BMNG watermask file.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 12-11-07, 16:36 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I found something interesting at this site page. http://rapidfire.sci.gsfc.nasa.gov/faq/ I'm referring to the questions about "What do the different band combinations mean?"
" How do I make true-color MODIS images?"
If I can find the original data used to create the BMNG series, I may be able to do something here about the missing water north of 60. I may even be able to create a 250 meter BMNG from more recent MODIS data. Looking into it.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Mon, 12-11-07, 21:19 GMT 
Offline

Joined: Thu, 18-10-07, 23:53 GMT
Posts: 16
I don't fully understand what you're talking about, but I do appreciate all the work. I'm currently using the SWBD_Watermask file to create new textures. We'll see how it goes.

*my 2 cents* :)

_________________
My PC: AMD Athlon X2 4400+; 2 GB of OCZ Platinum RAM; 320 GB SATA HDD; NVidia EVGA GeForce 7900GT KO, PCI-e, 512 MB, ForceWare ver. 163.71; Abit AN8 32X motherboard; 600 watt Kingwin Mach1 PSU; Windows XP Media Center SP2;


Top
 Profile  
 
 Post subject:
PostPosted: Tue, 13-11-07, 15:12 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I'm wondering if something should be done about this.
Here is an image. Notice the lakes in the upper right corner. Look a little too bright?Image
When I look at the specmap it shows shades of gray.

Image

Image

The original specmap has only 2 values, 0 and 255.

Image

Should the white pixels be given a max value less than 255?

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 15-11-07, 12:34 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I've been looking through the gdal source code to try and figure out what was done when I processed these files because after reading the man page for gdal_warp I noticed that it defaults to nearest neighbor resampling. I didn't use the program but there was resampling. I haven't found which resampling method was used with gdal_rasterize. The gdal_warp program has 4 resampling methods available. Nearest neighbor, bilinear, cubic, and cubic spline. Anyhow, I'm not sure if the highest quality map was produced. The gdal_rasterize program does not give an option for a resampling choice. I'm trying to trace the path through the files that gdal_rasterize is using but haven't found which resampling method it used or where to make a change to get it to use cubic or cubic spline if it is not using one of those methods already.

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 15-11-07, 19:52 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Here is the difference. The left side was run with gdal_rasterize and burned on a 3601x3601 file and then scaled down with cubic spline via gdalwarp. Gdalwarp also placed the file in the right spot on the world map. The right side was cropped out from the file I did already which had gdal_rasterize scale down the file in an unknown resample method. Which would be the best resample method for a 2 value image?

Image
Some pixels are gray in the cubic spline image. Some edges may be look smoother.
:?:

I was thinking of doing another run with this script.
Code:
#! /bin/sh
# make pngfile 3601x3601 template.png values=0
# make world file from 0 values in srtm and convert to geo-tiff
name=${1%.*}
name1=${1:2:8}
lon=${1:3:3}
lat=${1:7:2}
echo gdal_translate -a_ullr "-"${lon} $((lat + 1)) "-"$((lon - 1)) ${lat} template.png ${name1}.tif
gdal_rasterize -b 1 -burn 255 -l ${name1} -a FACC_code ${name}.shp ${name1}.tif
echo ${name1}
#gdalwarp -rcs ${name1}.tif world.tif

Trouble is that lon - 1 will not work. I get an error that says
Quote:
./batch-newrun1: line 8: 091: value too great for base (error token is "091")

If I change
Code:
$((lon -1))
to
Code:
${lon}
I get a line which looks like
Quote:
gdal_translate -a_ullr -091 39 -091 38 template.png w091n38n.tif
077100 - done.
w091n38n
gdal_translate -a_ullr -091 42 -091 41 template.png w091n41n.tif
077100 - done.
w091n41n
gdal_translate -a_ullr -091 40 -091 39 template.png w091n39n.tif
077100 - done.
w091n39n
gdal_translate -a_ullr -091 41 -091 40 template.png w091n40n.tif
077100 - done.
w091n40n

That second -091 needs to be -090 for the command to work.
I can't understand why (lat + 1) works and (lon - 1) doesn't :?
If I ever figure this out I'm thinking of running this in 2 runs, east and west on the 1 world file. Probably take about a week 24/7. Think it would be worth it?
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 15-11-07, 21:21 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4652
Location: Hamburg, Germany
cartrite,

with the naked eye, it seems pretty difficult to locate some essential differences between the left and the right displays.

But in general, cubic spline rescaling etc is significantly superior...

Bye fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 16-11-07, 11:41 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I fixed the script. It involved dealing with the leading zeros in the file names. Bash couldn't do math on integers with leading zeros. Here are some full 500 meter resolution images of the test area I did around 90-91w 38-40n.
First is the nearest neighbor.

Image

Bilinear

Image

Cubic

Image

Cubic Spline

Image

If you click on the images you'll notice there are more gray pixels in all but the nearest neighbor image. These gray pixels can all be converted to 0xff or left as is.

I still need to tweak the script. Among other things I want to have the ability to pause and resume it because the run will last days if not a week or more. A byproduct of this run is I will have 12000+ 1296k resolution ( 30 meter ) specmap tiles. That is probably be the best reason to do it. Some day they may come in handy. :wink:
cartrite


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 148 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6, 7 ... 10  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group