It is currently Sat, 19-08-17, 22:19 GMT

All times are UTC




Post new topic Reply to topic  [ 148 posts ]  Go to page 1, 2, 3, 4, 5 ... 10  Next
Author Message
 Post subject: New spec map needed?
PostPosted: Thu, 25-10-07, 18:31 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
I read a lot about alignment problems with the watermask specmap from snowy alaska here at the forum and I thought that someone was creating a spec map from the swbd files.
Anyhow, Last year I was trying to create a spec map by running an algorithm on srtm3 files that look for slopes no greater than 1 or less than 1 to extract rivers and lakes from the height files. I was having pretty good success with it until my computer's cpu fried. When I got the new one, I never started the task again. Is there any interest in this? The big problem is that the srtm3 files are 1201x1201 tiles and the srtm 84k topo map is too noisy to run such an algorithm on. I may be able to load these tiles into ISIS to create a less noisy height map that is 86400x43200 which will make extracting rivers and lakes possible. The only thing holding me back is I'm thinking that the new map will become just as noisy after it is scaled down to 84k.
cartrite


Top
 Profile  
 
 Post subject: Re: New spec map needed?
PostPosted: Thu, 25-10-07, 19:10 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4474
Location: Hamburg, Germany
cartrite wrote:
I read a lot about alignment problems with the watermask specmap from snowy alaska here at the forum and I thought that someone was creating a spec map from the swbd files.
Anyhow, Last year I was trying to create a spec map by running an algorithm on srtm3 files that look for slopes no greater than 1 or less than 1 to extract rivers and lakes from the height files. I was having pretty good success with it until my computer's cpu fried. When I got the new one, I never started the task again. Is there any interest in this? The big problem is that the srtm3 files are 1201x1201 tiles and the srtm 84k topo map is too noisy to run such an algorithm on. I may be able to load these tiles into ISIS to create a less noisy height map that is 86400x43200 which will make extracting rivers and lakes possible. The only thing holding me back is I'm thinking that the new map will become just as noisy after it is scaled down to 84k.
cartrite


cartrite,

it is quite clear that we urgently need a good 64k specmap replacement of the present official watermap. I reported elsewhere that some days ago, I wrote to the BMNG lead scientist Reto Stoeckli for advice, with whom I have communicated earlier. Unfortunately, he has not yet answered.

But clearly any other attempts that sound feasible should be looked into.

The main problem with the present watermap is NOT its alignment wrto the base map. It's the MISINTERPRETATION of water areas (lakes) where for sure there won't be any ;-) (deserts).

So let's discuss if you got some new ideas...

Cheers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 25-10-07, 19:54 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
This is the only file I could find from my efforts last time. It traced out the Delaware river without turning the southern part or New Jersey into an ocean which I know is very flat and surely could have been flagged as water. Deserts, I'm not sure.
The program looked at the nearest neighbor for equal values or values <1 or >1. Then it looked at it's neighbor. Then again. If all 3 tests passed the initial pixel was turned white or fffffff otherwise 000000 or some gray color.
You remember that. You gave me the idea.
Anyhow here is an initial test result from South Jersey.
I'm thinking of starting this now because I'm waiting until December for the next MROCTX data release to continue on with my mars project.

Image



cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 25-10-07, 20:06 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4474
Location: Hamburg, Germany
Yes, we had some intensive discussions about it. Could you post the code so I and or DW or others could play with it as well?

Cheers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 25-10-07, 20:43 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
I just tried to post some of it in my last post but got an error every time I hit the preview. I'll have to go through the files and figure out which ones were working correctly and zip them up and email them.
I think I had 3 runs For now, I can get this part in from run 1part1
Code:
            r = cell[y][x];
            gray[x] = 0x50;
            if (((y == 0 && x == 0) && (r == 0)) || ((y == 0 && x == 0) && (r == cell[y][x+1])) || ((y == 0) && (x > 0 && x < width) && ((r == cell[y][x-1]) || (r == cell[y][x+1]))))
                    gray[x] = 0xff;
            else if (((y == 0 && x == width) && (r == 0)) || ((y == 0 && x == width) && (r == cell[y][x-1])))
                    gray[x] = 0xff;
            else if (((y == height) && (x == 0) && (r == 0)) || ((y == height) && x == 0 && (r == cell[y-1][x])))
                    gray[x] = 0xff;
            else if (((y == 0) && (x == width) && (r == 0)) || ((y == 0) && (x == width) && (r == cell[y][x-1])))
                    gray[x] = 0xff;
            else if (((x == 0) && (r == 0)) || ((x == 0) && (r == cell[y][x+1])))
                    gray[x] = 0xff;
            else if (((x == width) && (r == 0)) || ((x == width) && (y != 0 || y != height) && (r == cell[y-1][x])) || ((x == width) && (y != 0 || y != height) && (r == cell[y+1][x])) || ((x == width) && (r == cell[y][x-1])))
                    gray[x] = 0xff;
This is run1 part2
Code:
                if (y < height && y > 0  && x > 0 && x < (width))
                {
                a = cell[y-1][x-1],
                b = cell[y-1][x],
                c = cell[y-1][x+1],
                d = cell[y][x-1],
                e = cell[y][x+1],
                f = cell[y+1][x-1],
                g = cell[y+1][x],
                h = cell[y+1][x+1];
      {
                if ((r == a) && ((r - 1) == h || r == h))
          if (k > 0 )
                       gray[x] = 0xff;
          else
             k++;
                else if ((r == b ) && ((r - 1) == g || r == g))
                    if (k > 0 )
                       gray[x] = 0xff;
          else
             k++;
                else if ((r == c ) && ((r - 1) == f || r == f))
                    if (k > 0 )
                       gray[x] = 0xff;
          else
             k++;
      else
          k = 0;
Then I write gray, width, 1, stdout
This is all I can post because I keep getting an error. It's probably not going to make much sense.
I'm going to go though my bash history on my older systems and see if I can figure out what I was doing with what files and zip them. Then I can email them to someone.
cartrite

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 25-10-07, 21:11 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
I think I got the right files zipped. Here is the command lines from the bash shell I used on the file above. Notice that I ran it several times with the same program. I'm think this was to stop non water areas from being selected as water.
Code:
g++ -O3 bmng-spec-part1b.cpp -o part1b
g++ -O3 bmng-spec-part3b.cpp -o part3b
g++ -O3 bmng-spec-part2b.cpp -o part2b
/home/cartrite/bin/spec-tool/phase7/part1b 1201 1201 < N39W075.hgt > /srv/disk3/dems/spec1.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1.bin > /srv/disk3/dems/spec1a.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1a.bin > /srv/disk3/dems/spec1.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1.bin > /srv/disk3/dems/spec1a.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1a.bin > /srv/disk3/dems/spec1.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1.bin > /srv/disk3/dems/spec1a.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1a.bin > /srv/disk3/dems/spec1.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1.bin > /srv/disk3/dems/spec1a.bin
/home/cartrite/bin/spec-tool/phase7/part2b 1201 1201 < spec1a.bin > /srv/disk3/dems/spec1.bin
/home/cartrite/bin/spec-tool/phase7/part3b 1201 1201 < spec1.bin > /srv/disk3/dems/spec1.pgm
Can I attach a zip file to a PM here?
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 25-10-07, 21:19 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4474
Location: Hamburg, Germany
Many thanks, cartrite.

I'll get into this as soon as time allows.

Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 25-10-07, 21:46 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
I just sent the files called works.zip to your email link at Shatters. You'll probably find them rather raw. I hope they were the right ones. I haven't had time to run them yet. I have to rebuild them for Suse 10.2 and find my srtm3 library on dvd. Somewhere buried in the closet no doubt.
cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Thu, 25-10-07, 22:29 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4474
Location: Hamburg, Germany
cartrite wrote:
I just sent the files called works.zip to your email link at Shatters. You'll probably find them rather raw. I hope they were the right ones. I haven't had time to run them yet. I have to rebuild them for Suse 10.2 and find my srtm3 library on dvd. Somewhere buried in the closet no doubt.
cartrite


Many thanks, cartrite,

both your mails arrived safely in Germany ;-) . Latest on the weekend, I'll dig myself into them. This is really an urgent homework.

Cheers,
Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 26-10-07, 0:06 GMT 
Offline
User avatar

Joined: Sat, 13-10-07, 18:18 GMT
Posts: 373
If you guys wish to test ANYTHING to do with specularity, then please
let me in on the testing...

As you may know, I'm having REAL problems with this issue right now.

Thanks, Bob

_________________
Brain-Dead Geezer Bob
Windows 8.1 Home Premium
AMD A8-5500 APU with Radeon HD Graphics
8.00 GB RAM, 64-bit Operating System
HP Hardware


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 26-10-07, 10:49 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
Hey Bob,
Not ready in this thread for testing yet. Still in the creation phase. Or about to start the creation phase. I'm sure there is plenty of testing on other fronts though. I think what you need to do is run the F-tex tools on a BMNG "month" and start looking for buggy's if there are any.

On another note I was wondering if it would simplify things if an algorithm can "clean up" the 84k map so that could be used instead of all those tiles from SRTM3. There seems to be minor fluctuations in the 84k map that are not present in the SRTM3 files. If these fluctuations can be "smoothed out" somehow, then the 84k map can be used to extract inland water data.

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 26-10-07, 13:50 GMT 
Offline
User avatar

Joined: Sat, 13-10-07, 18:18 GMT
Posts: 373
Cartrite,

I HAVE the latest tools package, but I'm just NOT quite ready to start
using them yet. I have a whole host of OTHER things to deal with right
now.

1. I'm trying to figure out this damned Vista O.S. (Getting there...)
2. I'm modifying my whole system to adapt it specifically for Celestia.
3. I'm supposed to be adding Wiki pages. Have done a couple, but need more.
4. I'm raising hell on the Celestia Forum in order to generate more discussion.
5. I'm creating a web site again. Have a single page now, but need MORE.
6. I'm trying to remember everything I forgot while I was away. Not easy...
7. I'm helping Selden in every way that I can. Numerous things.
8. I'm rewriting all of my tours and creating new ones for asteroids, comets.

Just SOME of the things I'm doing, but I WILL get there. Just give me a day
or two. Whew! :lol:

_________________
Brain-Dead Geezer Bob
Windows 8.1 Home Premium
AMD A8-5500 APU with Radeon HD Graphics
8.00 GB RAM, 64-bit Operating System
HP Hardware


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 26-10-07, 16:06 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
I haven't done one yet either. I'm planning on doing April. That would make my 3rd bmng I did. I also did October and July. Each one was done differently but they both correct the dark water areas along the coast lines. Before I do April, I would like to run them through the coastal water correction programs I wrote to do the July version. I would also like to create a spec map first instead of using my old specmap or the watermask specmap. My old specmap was created from the srtm 84k height file but only for the oceans. I did it last year. I think that's what the F-tex tools are doing too but I think they are also adding in data from the watermask file for inland waters.

The programs I sent Fridger actually do work on the SRTM3 files. I just rebuilt them on openSuse 10.2 and ran them on N39W075.hgt and N39W076.hgt to come up with a test file.

Image


I needed to run the part2b program a couple of more passes I think. That program searches for single white pixels and turns them black. My lame attempt to clean up the noise.
Here is a look at the history from bash. I combined the 2 pgm's with the gimp.
Code:
g++ -O3 bmng-spec-part1b.cpp -o part1b
g++ -O3 bmng-spec-part2b.cpp -o part2b
g++ -O3 bmng-spec-part3b.cpp -o part3b
/home/cartrite/specmap-work-area/works2/part1b 1201 1201 < N39W075.hgt > /home/cartrite/specmap-work-area/works2/spec1.bin
/home/cartrite/specmap-work-area/works2/part2b 1201 1201 < spec1.bin > /home/cartrite/specmap-work-area/works2/spec1a.bin
/home/cartrite/specmap-work-area/works2/part2b 1201 1201 < spec1a.bin > /home/cartrite/specmap-work-area/works2/spec1.bin
/home/cartrite/specmap-work-area/works2/part3b 1201 1201 < spec1.bin > /home/cartrite/specmap-work-area/works2/spec1.pgm
/home/cartrite/specmap-work-area/works2/part1b 1201 1201 < N39W076.hgt > /home/cartrite/specmap-work-area/works2/spec2.bin
/home/cartrite/specmap-work-area/works2/part2b 1201 1201 < spec2.bin > /home/cartrite/specmap-work-area/works2/spec2a.bin
/home/cartrite/specmap-work-area/works2/part2b 1201 1201 < spec2a.bin > /home/cartrite/specmap-work-area/works2/spec2.bin
/home/cartrite/specmap-work-area/works2/part3b 1201 1201 < spec2.bin > /home/cartrite/specmap-work-area/works2/spec2.pgm


I search my bmng level5 folder and found tx_18_8 and tx_18_9 and put them together. I then roughly cropped out a 364x182 section and combined it with the above spec map scaled down to the same size. Here are the results.

Image
+
Image
=
Image

cartrite


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 26-10-07, 18:17 GMT 
Offline
Site Admin
User avatar

Joined: Fri, 31-08-07, 7:01 GMT
Posts: 4474
Location: Hamburg, Germany
cartrite,

so it's Friday night over here. Weekend ahead ;-) .

I just compiled your 3 files part1b part2b and part3b and I read that they work also on SRTM3. That looks interesting. I get back to you once I understood the code.

But let me ask a question of principle here already:

Given elevation files and/or base tex files only, it seems impossible by any kind of algorithm to derive uniquely small areas of water notably in form of isolated lakes i.e that have no connection to other water areas (within the resolution!).

It is not at all surprising that the present watermask misidentificationes happened preferrably in desert areas that typically feature extended flat areas of sand. Without further info e.g. about vegetation colors in the neighborhood, it is simply impossible to uniquely distinguish flat sand from flat water...

Bye Fridger

_________________
Image


Top
 Profile  
 
 Post subject:
PostPosted: Fri, 26-10-07, 18:29 GMT 
Offline
User avatar

Joined: Thu, 25-10-07, 15:20 GMT
Posts: 991
Location: NE PA, USA
Fridger,
I can't remember all, it's been over a year now since I read the paper the describing how the swbd, srtm1, and srtm3 were created but I think I can remember that lakes were isolated to a "forgotten" threshold. It was larger than 180 meters or so. Anything smaller than the threshold was not altered. These areas were given a constant value = -1 the surrounding shoreline values. That is why the srtm3 files work. I don't think the srtm 84k file was done to that level. And that's why I could not get any program too come close to working on that file.

I'm not sure what was going on with the watermask file. I think I remember looking at the amazon river basin and many of the tributaries were offset by 30 to 50 pixels or more in reference to the bmng file.

I was unaware that flat desert areas were put into the watermask as water though. I never got around to doing a desert area with these programs. That should be a starting point to see if these programs do the same thing. Turn deserts into water that is. I was doing South America when my computer fried a while back. That was the last time I worked on this.

There is one area near your neck of the woods that will probably be problematic. Maybe you corrected that one though? Holland and Denmark areas near the coast are below sea level but dry because of dikes and sea walls. Aren't these areas being put in the spec map as water when treating 0 values as water from the 84k srtm map?

Edit: The paper I was referring to is called SRTM_edit_rules.pdf. I can't remember what the link was were I downloaded it from. I still have a copy though.

cartrite


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 148 posts ]  Go to page 1, 2, 3, 4, 5 ... 10  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group