kml_filter.py

Found a problem? Tell us about it! Topics-per-issue, please search before posting.

16 posts • Page 1 of 2

Postby neoskyy@pm.me » Mon Mar 25, 2019 2:43 am

Hello,

I am attempting to use the provided kml_filter.py script, but there are some errors that are preventing the script from producing any output:

Code: Select all

PS C:\Python27> python .\kml_filter.py -o python_sorted.kml .\WigleWifi_20190317222843.kml No handlers could be found for logger "fastkml.config" .\WigleWifi_20190317222843.kml: 1 <generator object features at 0x0000000004384048> Traceback (most recent call last): File ".\kml_filter.py", line 193, in <module> converter.main() File ".\kml_filter.py", line 70, in main self.parse(filename) File ".\kml_filter.py", line 144, in parse if "Encryption: None" in placemark.description: TypeError: argument of type 'NoneType' is not iterable
Any thoughts?

Postby arkasha » Mon Mar 25, 2019 5:30 pm

what's the filesize of the input file?

if you load it in google earth, what do you see?

Cheers, and thanks for your interest!

Postby neoskyy@pm.me » Mon Mar 25, 2019 7:17 pm

Hi,

The size of the .kml is 8,021KB, 54,859 lines, according to the WiGLE uploads page. This is the transaction ID: 20190323-00024.

Loading the .kml into Google Earth works and I see all the markers, though it brings the app to its knees and hardly functions. I do not see how to easily sort by encryption types in Google Earth either, hence my interest in the kml_filter.py script.

As a side note, I noticed looking at the .sqlite database and the CSV that there is no columnar data for whether or not a network is open? I would think that whether or not a network has encrypted password protection would be an important data point of interest, just as much as the encryption type (WEP/WPA etc.).

Thanks for your quick reply and help.

Postby arkasha » Mon Mar 25, 2019 11:11 pm

interesting - I gave it a try in my virtual environment and I get a successful run with your file! (yeah, I'm an admin, I can do that)

try blowing away your virtual environment/re-creating, and I'll do the same?

Cheers,

-ark

Postby neoskyy@pm.me » Tue Mar 26, 2019 1:49 am

Trying now in Debian, Python version 2.7.13:

root@debian:/home/laptop/Downloads# python kml_filter.py -o python_filtered.txt WigleWifi_20190317222843.kml
WigleWifi_20190317222843.kml: 1
<generator object features at 0x7f24f4aca370>
Traceback (most recent call last):
File "kml_filter.py", line 193, in <module>
converter.main()
File "kml_filter.py", line 70, in main
self.parse(filename)
File "kml_filter.py", line 144, in parse
if "Encryption: None" in placemark.description:
TypeError: argument of type 'NoneType' is not iterable

Same error.

Postby arkasha » Tue Mar 26, 2019 2:14 pm

question: how are you accessing these KML files? (how/where are you downloading them?)

Postby arkasha » Tue Mar 26, 2019 2:22 pm

specifically, the filter tool was written to process the KML format generated by the "link" and "view" tools in the "Uploads" tab of the app, or by downloading using the transid link on the website uploads page; the export run as KML button on the database tab uses a different format.

Postby neoskyy@pm.me » Tue Mar 26, 2019 6:13 pm

I see. I might actually be trying to use the the .kml produced by the 'Export' feature from the app. I will try the script again, but this time against the .kml downloaded from the Uploads links here on WiGLE.net

Postby arkasha » Tue Mar 26, 2019 8:56 pm

awesome - we actually should evaluate the differences between the formats - these grew organically, but there probably isn't a great reason for the difference.

Cheers,

-ark

Postby neoskyy@pm.me » Wed Mar 27, 2019 7:24 am

@arkasha your assumption was correct, I was using the .kml produced by and exported from the WiGLE Android app and that was the issue. After downloading the .kml from the Uploads page here on WiGLE.net, the kml_filter.py script ran correctly. Here was my output:

unencrypted: 23
unknown: 2355
wep: 214
wpa: 17179
wpa2: 17119
wpa3: 0

2 questions: first, I can see that there were 23 unencrypted networks according to the script's output, but looking at the .sqlite and CSV exports, there appears to be no columnar data or indication of a network being unencrypted? As I need to run SQL queries against this data set, it would be helpful to know where unencrypted networks are indicated. Second question, is it a bug that wpa and wpa2 have the same number of results?

Postby arkasha » Wed Mar 27, 2019 5:49 pm

1.) the lack of any of those other encryption indicators is the "unencrypted" status. I can't urge you strongly enough to work with the CSV rather than the KML, since the KML just a quick visualization hack on top of the CSV data, and isn't structured for database-style access (the info is just shoved into the "bubble")

2.) that's doing string matching - it may be a bug, or all the networks in your file may contain both strings (usually, the capabilities array contains [WPA],[WPA2] if it's a "WPA2" network.

Cheers,

-ark and the WiGLE team

Postby neoskyy@pm.me » Thu Mar 28, 2019 9:32 pm

@arkasha in regards to your response #1, that was actually the confusing part for me. For instance, when I examine the .kml for a network called Phillips5G-Guest, I see this:

Code: Select all

Encryption: None Time: 2019-03-16T15:15:51.000-07:00 Signal: -81.0 Accuracy: 4.0 </description> <styleUrl>#highConfidence</styleUrl> <Point> <coordinates>-111.90319061,40.31055069</coordinates> </Point> </Placemark> <Placemark> <name>Phillips5G-Guest</name> <open>1</open> <description>Network ID: 52:6A:03:AF:D9:7B
There is a clear indicator that no encryption is being used. However, looking at the same network in the CSV:

52:6a:03:af:d9:7b Phillips5G-Guest [WPA2-PSK-CCMP+TKIP][WPA-PSK-CCMP+TKIP][ESS]

The AuthMode column shows that WPA2 is being used, and there seems to be no indication that no encryption is being used. The same goes for the .sqlite, there is no indication that the network is open or unencrypted, or Encryption: None.

Postby arkasha » Thu Mar 28, 2019 10:50 pm

Without being to produce the exact circumstances, I'd recommend using the SQLite or CSV files locally, or the query API online for your purposes; the KML files are intended for visualization rather definitive database-like access, and it's possible that they failed to capture the status of this network correctly. Running spot checks on my own KML exports, I see an Encryption status inside each

Code: Select all

<description>...</description>
block.

The snippet included in your example, you seem to have included the tail-end of the previous placemark's description (ending with

Code: Select all

</Point> </Placemark>
) and the beginning of the Phillips5G-Guest placemark - however we stop at the end of network ID - this would be followed by a newline and then the Encryption status for that network in the uploads page export format.

Postby arkasha » Sun Mar 31, 2019 12:20 am

on-site KML export has been updated, as has the kml_filter script to bring it into alignment with client export. We'll update client KML export further in the next release in order to include encryption status as well as BT and Cell export. They'll still contain some disparate fields - there are elements of data that only exist on the client or server - but they'll be parsable by the same tools from now on.

Thanks for the heads-up.

Postby neoskyy@pm.me » Mon Apr 01, 2019 3:55 am

@arkasha That is great to hear that those changes will be made. After looking at the data, I believe the results for WPA and WPA2 are indeed bugged in the script. I used this SQL (querying the .sqlite exported from the Android WiGLE client) to get the proper totals for WPA vs WPA2:

Code: Select all

SELECT * FROM network WHERE capabilities LIKE '[WPA-%';
Thank you again.

16 posts • Page 1 of 2

Return to “Bugs”

Who is online

Users browsing this forum: melbournefan and 1 guest