WiGLE.net

That link seems to be showing at least a couple of transactions processed from today (19th) processed and a few from yesterday. I've got an upload posted from the day before that languishing in the queue which apparently hasn't advanced in 30 hours. Also the two posts before that which have been processed aren't reflected in the stats yet. Isn't processing meant to be in the order submitted?

processing is 100% in the order that we get the files committed.
the stats page is a batch aggregate, and not a queue status monitor.

Looks more like it is being done in alphabetic username order. Some of my queued files haven't moved up at all for 3 days now, yet some users with names beginning with a number or a, b, c have been processed more recently.

pretend for a second that i can read the source code to the system that i've been helping to operate for a dozen years,
and trust me on this one.

then go back and re-read my last post.

OK. I have read it several times and am sure you understand your system. Please explain why my upload 20140217-00265 was stuck at #386 in the queue until a few hours ago, not moving for a full 3 days while one or more batches were aggregated into the stats containing:
ccie4526 last post 19-Feb-2014
CaptSalty last post 19-Feb-2014
catflop last post 19-Feb-2014
Camager last post 19-Feb-2014
bobzilla last post 19-Feb-2014
calltom71 last post 19-Feb-2014
astronom40 last post 19-Feb-2014
anonymous last post 19-Feb-2014
arkasha last post 18-Feb-2014
blitz last post 18-Feb-2014
...
and 20 more with a last post of 18-Feb-2014

Noticeably, there are no usernames in the list of 30 that start further down the alphabet than C. Even if the batches are aggregated into the stats are made up in alphabetical order and D-Z haven't been processed yet, the last post date implies that uploads from the 18th and 19th have been processed, while one or more uploads from the 17th are still shown way down the queue. Please note my use of the word 'implies' in the previous statement - I am not saying it IS a fact that processing out of date order is taking place, I'm merely saying that it LOOKS LIKE processing out of order is taking place, as I stated in my previous post. I'm not demanding explanations, fixes, etc., just politely asking for clarification.

fair enough.
the aggregate stats page is only *actually* up to date when the files to be processed = 0.
it at no point reflects the processing order in the queue, especially when there is an exciting backlog of work.

if /you/ had a transid that was processed/completed out of order: something horrible has broken.
otherwise, if your local partial order is maintained, that's just how the batching works out.

quote- from TROMOS

OK. I have read it several times and am sure you understand your system. Please explain why my upload 20140217-00265 was stuck at #386 in the queue until a few hours ago, not moving for a full 3 days while one or more batches were aggregated into the stats containing:
ccie4526 last post 19-Feb-2014
CaptSalty last post 19-Feb-2014
---
The file that posted 19 Feb was actually uploaded 16 Feb.

Likewise I currently am #12-#16 in the queue and it has not moved in the last 6 hours. I would hope there is a dead-man timer that would kill a file that was taking that long to process.

Still #12 to #16 in the queue.

Looks like anonymous suffered an integer rollover.

there was something furry stuck in the gears; i've given them (the gears) a good poking stick^sm.
the stats page is still not going to be accurate until the queue has been worked through.

there was something furry stuck in the gears; i've given them (the gears) a good poking stick^sm.
the stats page is still not going to be accurate until the queue has been worked through.

Looks like you need to get the stick out again, stuck at #74 for 4+ hours.

Wait...wigle has an IRC channel?

since forever.
you must not ever read the front page of wigle.net, i'm guessing?

Just not very observant.

We run the files in the queue, then the users for the files (so the common case of a user uploading multiple files doesn't have us running stats on a user multiple times). The user stats will take account everything uploaded for the first/last post timestamp, even if it hasn't been processed yet (eventually consistent). Lots of batching everywhere for throughput, which no one notices except when things get backed up.

Its hitting on all cylinders now, only 152 seconds to process a 20M .gpsxml file.. and zero in the queue.