It is currently Mon, 20-11-17, 7:45 GMT

All times are UTC




Post new topic Reply to topic  [ 148 posts ]  Go to page Previous  1 ... 6, 7, 8, 9, 10  Next
Author Message
 Post subject:
PostPosted: Thu, 16-05-13, 2:51 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I used cubic spline reduction on those above runs yesterday and this morning. I tried cubic reduction this afternoon but I also tried to map the raw files and change the "1's" to 255 at the end. That had it's problems too. A large black area from top to bottom and across the top.

I just got done with a run with cubic reduction and the images were processed by changing the value "1" to 255 before mapping. Here is the scripts I wrote to do this. I ran this first
Code:
cd modis-gz-files
for %%I in (*.gz) do gzip -dc < %%I > ..\modis-tif-files\%%~nI

Then after I separated the files by their four different dimensions, I ran this.
Code:
cd modis-tif-files\5761_7681
for %%I in (*.tif) do gdal_translate -of png %%I ..\..\png-files\5761_7681\%%~nI.png
cd ..\..\png-files\5761_7681
move *.xml ..\..\modified-png-files
cd ..\..\modis-tif-files\5760_7681
for %%I in (*.tif) do gdal_translate -of png %%I ..\..\png-files\5760_7681\%%~nI.png
cd ..\..\png-files\5760_7681
move *.xml ..\..\modified-png-files
cd ..\..\modis-tif-files\5761_9601
for %%I in (*.tif) do gdal_translate -of png %%I ..\..\png-files\5761_9601\%%~nI.png
cd ..\..\png-files\5761_9601
move *.xml ..\..\modified-png-files
cd ..\..\modis-tif-files\5760_9601
for %%I in (*.tif) do gdal_translate -of png %%I ..\..\png-files\5760_9601\%%~nI.png
cd ..\..\png-files\5760_9601
move *.xml ..\..\modified-png-files
cd ..\png-files\5761_7681
for %%I in (*.png) do png2bin < %%I > ..\..\bin-files\5761_7681\%%~nI.bin
cd ..\5760_7681
for %%I in (*.png) do png2bin < %%I > ..\..\bin-files\5760_7681\%%~nI.bin
cd ..\5761_9601
for %%I in (*.png) do png2bin < %%I > ..\..\bin-files\5761_9601\%%~nI.bin
cd ..\5760_9601
for %%I in (*.png) do png2bin < %%I > ..\..\bin-files\5760_9601\%%~nI.bin
cd ..\..\bin-files\5761_7681
for %%I in (*.bin) do invert-modis 5761 7861 %%I > ..\..\modified-bin-files\5761_7681\%%~nI.bin
cd ..\5760_7681
for %%I in (*.bin) do invert-modis 5760 7861 %%I > ..\..\modified-bin-files\5760_7681\%%~nI.bin
cd ..\5761_9601
for %%I in (*.bin) do invert-modis 5761 9601 %%I > ..\..\modified-bin-files\5761_9601\%%~nI.bin
cd ..\5760_9601
for %%I in (*.bin) do invert-modis 5760 9601 %%I > ..\..\modified-bin-files\5760_9601\%%~nI.bin
cd ..\..\modified-bin-files\5761_7681
for %%I in (*.bin) do bin2png 1 5761 7681 < %%I > ..\..\modified-png-files\%%~nI.png
cd ..\5760_7681
for %%I in (*.bin) do bin2png 1 5760 7681 < %%I > ..\..\modified-png-files\%%~nI.png
cd ..\5761_9601
for %%I in (*.bin) do bin2png 1 5761 9601 < %%I > ..\..\modified-png-files\%%~nI.png
cd ..\5760_9601
for %%I in (*.bin) do bin2png 1 5760 9601 < %%I > ..\..\modified-png-files\%%~nI.png
cd ..\..\modified-png-files
for %%I in (*.png) do gdal_translate -of GTiff %%I ..\modified-tif-files\%%~nI.tif
cd ..\
bin2png 1 86400 43200 < landmask.bin > landmask.png
gdal_translate -a_ullr -180 90 180 -90 landmask.png landmask.tif
cd modified-tif-files
for %%I in (*.tif) do gdalwarp -r cubic %%I ..\landmask.tif
cd..\
gdal_translate -of png landmask.tif landmask.png
png2bin < landmask.png > specmap.bin
tx2half 1 86400 43200 < specmap.bin > 32k.bin
tx2half 1 43200 21600 < 32k.bin > 16k.bin
tx2half 1 21600 10800 < 16k.bin > 8k.bin
bin2png 1 21600 10800 < 16k.bin > 16k.png
bin2png 1 10800 5400 < 8k.bin > 8k.png


And here was the result.

Image

I did map this to the file I created long ago and is still available here at CM.

I am going to try one more run using a new resampling method in Gdal called "mode". Apparently it samples the surrounding area and uses the value that there is most of. I'm hoping that will keep the values at 0 and 255. This has a lot of light gray and dark gray pixels. Most in Northern North America. Getting there. :D
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 16-05-13, 10:28 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4508
Location: Hamburg, Germany
So what's your judgement about the resulting quality and artefacts compared to your original spec map??

Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 16-05-13, 10:56 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I didn't get a chance yet to really look at quality issues. Overall, I was looking at obvious defects in processing the data. I'm also scanning the intermediate files with a hex editor to see if correct values are being generated. With these files, I should only see 3 values. Zero, one, and 255.

There may be a bug with gdalwarp when using cubic spline as reduction option. I think this is what produced all the gray blocks I noticed in a former run. The latest run used the cubic method.

I still see a lot of noise between 60 and 90 north. North America has the most. Gonna try this new method of reduction called "mode" Ultimately, I'd like to produce a file that has 2 values. Zero and 255.

In the end, to finish, I will probably use this new data to only supplement the SWBD map. Right now, I'm overwriting it. If I get my old computer running again, or just retrieve the data from it's hard drives, I have the original 90 meter files that were used to create the SWBD map that is available here. These were generated while creating the SWBD map. Then I can create a 250 meter map. This new data uses the same data I used for 60 n to 60 s. But the map here is only 500 meter.

After further exploring the source for this new data, http://glcf.umd.edu/data/watermask/ ,I found a lot more docs. Here is a citation for this new data I would like to post as well.
Quote:
Authors: Carroll, M., Townshend, J., DiMiceli, C., Noojipady, P., Sohlberg, R. 2009.
MOD44W MODIS Water Mask Collection 5.
A New Global Raster Water Mask at 250 Meter Resolution. International Journal of Digital Earth.
University of Maryland, College Park, Maryland 2000 - 2009
University of Maryland, Department of Geography and NASA; use is free to all if acknowledgement is made. UMD holds ultimate copyright.
Source for this data set is the Global Land Cover Facility, www.landcover.org.
http://glcf.umd.edu/data/watermask/description.shtml
http://glcf.umd.edu/data/watermask/ackn ... ents.shtml
http://glcf.umd.edu/library/guide/techg ... ermask.pdf
This citation also contains links that I would like to go over in detail. I would like to understand how they extracted this data above 60 north.

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 17-05-13, 14:21 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I finished a run using the "mode" method of resampling while running gdalwarp and it produced a slightly better file but I think I took this as far as I could for now. After seeing the results of the last run, I tried to "clean up" the file produced by the cubic method. Unfortunately, I started running into problems. The cleaning method was producing files with all zero's. I can still play around with that idea but I ran into another problem.

I tried to modify "Watermap.cpp" to do the cleaning. But when I try to use stdin on only one file, I don't think I have the memory. The program runs about half way and crashes. I also tried to use the original Watermap to combine my old file with the new one but I get the same result. It processes about half of the file and stops.

If you want, I can try to upload the compressed files from my last 2 runs and then you can take a look. The cubic file compressed to about 35 mb but the mode file compressed to about 12 like the original.

I did examine both files with a hex editor. In the "cubic" file there were many different values. The "mode" file produced only 2 values, zero and 255 as expected. But this may not be accurate. I need to find out what this is actually doing. I'm gonna download gdal's source and see if I can gain some understanding on this issue. Bye for now.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 17-05-13, 20:05 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
My last post was deleted :shock: :cry: I see the pics were still in the gallery if I should repost.Let me know.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 17-05-13, 22:13 GMT 
Offline
Site Admin
User avatar

Joined: Thu, 30-08-07, 22:52 GMT
Posts: 2726
Location: France, South, not far from Montpellier
Stephen, your post was probably lost after the crash of this afternoon. Unfortunately we cannot do much to retrieve your post, so be free to repost if you want to.


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 17-05-13, 23:14 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
ElChristou wrote:
Stephen, your post was probably lost after the crash of this afternoon. Unfortunately we cannot do much to retrieve your post, so be free to repost if you want to.


I did have problems connecting this afternoon. Didn't realize it was a crash.

I just got confused by the results I achieved while trying to clean some of the " noise" in one of the files. This is the code that was actually used. But I thought it was a typo.
Code:
        char a = 255;
        char b = 0;
        char c = 240;

///                if (h[i] == a)
                    hspec[i] = 255;
                else if (h[i] == b)
                    hspec[i] = 0;
                else if (h[i] >= c)
                    hspec[i] = 0;
I thought the last line should have read
Code:
 hspec[i] = 255;

The values I saw in the hex editor was "40" "80" "6c". I thought these would be less than 240 or "ff". But all of these were turned to zeros.

Anyhow, I posted the results from the runs.

Results from the cubic run before running the above code.


Image


Results from cubic run after running above code.

Image


And the results from using the "mode" option.

Image

I found it impossible to use the program watermap.cpp to combine the northern area with the map I did years ago. The program gets to the same point. About row 23552 or so. Then It crashes. And I fear I may have degraded the data between 60n and 60 south while "cleaning". I didn't clean the "mode" run. Anyhow, I've been running Celestia to find out what it looks like at 65k.

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 18-05-13, 2:44 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Well it wasn't impossible to combine my old map with the new one. I had to read both files using ifstream.

Anyhow, a lot of this noise I thought I was getting rid of was actually on the old map too. In fact the images I posted are not what was in question. That's a little further north. I guess I'll have to take a good look at what was done and see what is what. :? That will take time. :|
Well signing off for now. Later.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 18-05-13, 10:51 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
t00fri wrote:
So what's your judgement about the resulting quality and artefacts compared to your original spec map??

Fridger


So far, I produced a number of files and each one is noticeably different. Putting sample 16k files together in a folder and flipping thru them in a photo viewer, it is quite obvious. Thing is, the source data has been thru much rigorous testing and review. I'm not sure how to ensure their standards are in tact without artifacts introduced by the processing I did.

But............

I'd like to compare my old map with:
1. The output from the cubic run untouched. 35mb
2. The output from the cubic run cleaned. 12mb
3. The output from combining both the old map and both above. 13.5mb
4 The output from the mode run. 12 mb
I'd like to do 3 more runs.
One with nearest neighbor.
The second, I wrote a simple program that creates a 128k blank template.
This run will use gdalwarp but since the data will have the same resolution, there shouldn't be any resampling issues. To be sure, I'd like to compare that with a run with gdal_merge. This a python script that doesn't resize only mosaics.

A question: Is it possible to use txtiles to produce larger tiles? 4k?
That way it will be easier to isolate possible areas of concern with Celestia.

In the meantime, if you would like a copy to play with, PM me a location to send it. Later.

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 18-05-13, 19:39 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
Anyone ever hear the phrase
Quote:
Keep it simple stupid? :wink:

:D
Seems that all the work I did on this can be summarized to these 2 commands.
Code:
dir /b /s *.tif > file_list.txt
gdal_merge.py -v -o ..\128k-earthspec.tif -ul_lr -180 90 180 -90 -init 255 --optfile file_list.txt

Now, I modified the original *.tif files I downloaded by changing the "value = 255" to "value = 0" and the "value = 1" to "value = 255", so ...... This does need to be done as well. Running the above commands creates a 128k specmap with a terminal in the working directory. :shock: :lol: :wink:

After a quick look at the 21600x10800 level, this is a complete file except for Antarctica. The part from California that was missing in one of my first runs is in this one. An ERROR :shock:

I didn't finish reading this pdf file, http://glcf.umd.edu/library/guide/techg ... ermask.pdf , but this is probably a better file then the one I created. Just need to add in Antarctica. That's been done with the SRTM height file. :) EDIT This may be worth looking into.
http://nsidc.org/data/moa/

From what I seen running Celestia, what may be permanent water can still be covered by tree cover and not look right. So............ Time for a few drinks. :D

cartrite :shock: :D


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 25-05-13, 19:30 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
I'm wondering what should be the best method of producing a specmap with the latest data I came across?

I am thinking that the best map with the most up to date data on planetary water systems is the MOD44W files. But this shows all water that does not account for it being visible from space. A lot of this water is either infested with weeds or under tree cover and not visible from space. Thus it does not show up on the BMNG files.

I'm gonna to try and produce a file that will only show water systems that are visible from space.

Any thoughts?

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 25-05-13, 19:57 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 21:55 GMT
Posts: 773
Location: N 42.38846 W 83.45456
A question ?

how dose this different data set line up with the blue marble2 monthly maps

i am using the June map
ftp://neo.sci.gsfc.nasa.gov/bluemarble/bmng/world_500m/

and the landmask
ftp://neo.sci.gsfc.nasa.gov/bluemarble/ ... dmask_new/
is off by a few px
Hawaii is a good example

Quote:
I'm gonna to try and produce a file that will only show water systems that are visible from space.


probably the same as i am
The night lights needed a water mask
I used the June BMNG and created a partial 0 and 255 water mask


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 25-05-13, 20:57 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
John Van Vliet wrote:
A question ?

how dose this different data set line up with the blue marble2 monthly maps

The MOD44W files produce virtually the same file as the map I created with the SWBD files. Both use the same data. The MOD44W files are probably an improvement though. There are probably some areas that are not in alignment but most of the map lines up perfectly with BMNG maps that I am using.
Quote:
I was using the June BMNG from here.
http://mirrors.arsc.edu/nasa/world_big/ to do comparisons.
Quote:

and the landmask
ftp://neo.sci.gsfc.nasa.gov/bluemarble/ ... dmask_new/
is off by a few px
Hawaii is a good example

Haven't used this map.

Quote:
I'm gonna to try and produce a file that will only show water systems that are visible from space.


Quote:
probably the same as i am
The night lights needed a water mask
I used the June BMNG and created a partial 0 and 255 water mask

If you want, PM me with an email address and the format you need ( .png .bin etc. )and I'll send the file from MOD44W. This will be the one produced from the gdal_merge command. I think this to be most accurate.

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sat, 25-05-13, 22:50 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 992
Location: NE PA, USA
As luck would have it, Hawaii is in view when I start Celestia. Here is the level5 image that is displayed. There are a couple of artifacts but the alignment loos good.

Image

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Sun, 26-05-13, 0:08 GMT 
Offline
User avatar

Joined: Tue, 04-09-07, 21:55 GMT
Posts: 773
Location: N 42.38846 W 83.45456
looks nice

_________________
"I don't pitch Linux to my friends, I let Microsoft do that for me."
Using OpenSUSE 42.1 & Scientific Linux 6.7


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 148 posts ]  Go to page Previous  1 ... 6, 7, 8, 9, 10  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group