db maintenance, file processing off

Notes regarding project status and info

19 posts • Page 1 of 2

Postby bobzilla » Sat Jul 21, 2007 5:20 pm

wanted to let folks know that file processing will be off for much of this weekend for some db maintenance. feel free to keep uploading files, they will all get processed afterwards. we apologize for the inconvenience.
wanted to let folks know that file processing will be off for much of this weekend for some db maintenance. feel free to keep uploading files, they will all get processed afterwards. we apologize for the inconvenience.
Hey just curious, but what about the Web Maps? I see the file processing was turned back on yesterday, but the Web Maps are still down?

Postby bobzilla » Mon Jul 23, 2007 1:55 pm

thanks, i'm spinning that up now

Postby i_do_dew » Mon Jul 23, 2007 3:02 pm

Looks like the world and Europe maps need to be re-genned as well, or maybe just their front page thumbs.

Of course, like anything so much to do and only so many hampsters. I would suggest rabbits but it is harder getting useful work out of those because all they do is re-produce.

Postby uhtu » Mon Jul 23, 2007 8:57 pm

rabbits eh. i'll pass it on to the interns!
they're being directed to re-create those maps just as soon as the mapserver has cleared the planet.

Postby AZ_WD'er » Tue Aug 05, 2008 2:38 am

Looks like the world and Europe maps need to be re-genned as well, or maybe just their front page thumbs.

Of course, like anything so much to do and only so many hampsters. I would suggest rabbits but it is harder getting useful work out of those because all they do is re-produce.
Looks like the rabbits got REAL busy and reproduced a bit too fast. 800+ files in que and none processed in at least 20+ hours. Must be Party Time Rabbit Style. :)

Postby uhtu » Tue Aug 05, 2008 11:49 am

on the contrary, they have been processing pretty much continuously during that time (and are still going); the
dark magic that updates the processed/remaining dosen't get tweaked as often as would be handy to assuage concerns.

Postby dagaroth » Tue Aug 05, 2008 1:35 pm

lol, whoops, I should have read before I uploaded, I just dumped about 400 files up there too :P

Sorry to anyone that gets in behind me :(

Postby dagaroth » Tue Aug 05, 2008 7:06 pm

actually, processing seems slow, it's only moved about 60-70 files in 5 hours

Postby Blaze312 » Thu Aug 07, 2008 7:57 pm

Dammit...

I got stuck behind you. :-D

I am queued 440 now.

Just a question dagaroth, why so many files?

I think im beginning to finally understand this:
14,759,557 points from 895,974,862 unique observations.
I always wondered why there was almost a billion observations and less then 15 million actual points.

If i did my math correctly we are averaging 1 new point every 60 files upload. Why would this be? I mean i've often uploaded the same points as part of a stumble upload, but i almost always get at least 10 new points.

Sorry to go on on a rant... Just curious....

Postby uhtu » Thu Aug 07, 2008 8:49 pm

networks are (network)

observations are (network,lat,lon)

so you will always have at least as many observations as networks, and usually many many more.

Postby whitedice » Thu Aug 07, 2008 8:53 pm

Each file has multiple locations for 1 point. The *.gps file from kismet is exactly what it is... a file filled with gps locations continuously as you are moving, and recording what is at each point. I'm roughly guessing at that.

I have uploaded files in the past that have taken 70+ minutes to individually calculate. Imagine driving for 8 hours, and recording 13,000 points. Out of the 13,000, getting only 2,000 new. Lots of database searching, and trangulating. It wouldn't surprise me that the bigger the database gets, it will just take longer in general.

I usually only have about 20-30 files for a weeks worth of collecting while I'm driving

Postby dagaroth » Sat Aug 09, 2008 1:04 am

actually, I went back later and looked, it was 227.

I wardrive every day. If I'm in the car, my rig is on. Usually it's just to/from work, or to/from my kids school or daycamp, so most of my logs are less than 20 miles.

On actual wardrives, I plan ahead using google maps and google earth to maximize my routes, and I plan in stop points to shut down kismet and start a new logfile. I have a problem with my setup where if I run for more than an hour or so, everything locks, and I dont log anything, resetting before I hit an hour seems to fix that. I think it's my rig, I just need time/money to build a new one.

I'd also gotten lazy and didnt dump my stuff onto my USB stick in over a month :P

8 hour wardrives? damn man, are you like a courier or something? I cant actually wardrive for more than an hour or so without getting bored :P With the exception of the long family trips we take, or my occasional trip to pick up something I got off ebay, most of my planned wardrives are around an hour :)

Postby dagaroth » Sat Aug 09, 2008 1:06 am

and I just realized you're in Cinci whitedice.. are you the reason why the points I got in cleveland and medina where so low last summer :P

Postby whitedice » Sat Aug 09, 2008 6:19 am

8 hour wardrives? damn man, are you like a courier or something? I cant actually wardrive for more than an hour or so without getting bored :P With the exception of the long family trips we take, or my occasional trip to pick up something I got off ebay, most of my planned wardrives are around an hour :)
My job until today (i just quit to go back to school) was driving a semi around the Eastern US. I've logged 128,000 miles in 15 months.

To give you an idea here is one file, in my uploads list.
4344 seconds (72.4 minutes)
Whitedice-Jul-09-2008-1.csv
2,465,968 bytes
8,568 File lines
1,002 new networks w/ gps
1,003 total new discovered networks
8,567 total networks in file
8,567 total new locations

That particular file, I drove around Raleigh, NC picking up a load that went down near Charleston, SC. I turn on kismet, and did my job, and when I stop to sleep, I turned it off.

Someone has done a bunch to the east of cleveland out near 271. I drove downtown, and some major roads and universities around the area, and most of the stuff to the west is me (between Sandusky and Cleveland). It isn't all me, but a lot of it is. I did that before I started driving a truck with my SUV setup.

I'm in the process of creating an animated gif of my weekly additions while working. Kind of neat to see... In my opinion.

19 posts • Page 1 of 2

Return to “WiGLE News”

Who is online

Users browsing this forum: No registered users and 3 guests