Replace CSV with more efficient Parquet file format or add Parquet as second file format.

Suggestions for WiGLE/JiGLE/DiGLE

3 posts • Page 1 of 1
I'm playing with Jupyter Notebooks / Pandas and recently discovered the Parquet file format.
It has many advantages over CSV.
It would be nice to get the data from the site in bulk using that format.
As data is stored column by column instead of row by row it's more compressible, faster to load, faster to process.
Image
As the CSV format is a documented standard ( https://api.wigle.net/csvFormat.html ) shared between WiGLE and a number of apps and integrations, as well as much easier to interpret and debug visually, we're extremely unlikely to replace it.
We'll keep Parquet in mind as need dictates, although we see very little performance overhead from CSV loading even at our maximum upload file sizes. As for "getting" data from the site, we use structured JSON records, not CSV for most data export tasks.
big advantage of csv is its simplicity, every text editor can open it.
Yes its not the most space saving format. But for that you can just drop it into a zip file.
Another thing for me its the non binary content, no virus scanner or mail rule will block a csv.

For me (as ive written my own csv generators in python for wigle) i hope csv stays as upload format :)
Image

3 posts • Page 1 of 1

Return to “WiGLE Project Suggestions”

Who is online

Users browsing this forum: No registered users and 11 guests