Geocoded PostCode Database Leaked Online

NickG · Posted: Thu Sep 17, 2009 10:38 am Post subject:

I'm not sure this is the real Royal Mail database. It seems to be incomplete, and some (but not all) postcodes seem to be rounded to the nearest 100m grid (common in cheaper datasets used to implement "find my nearest store" type features). Perhaps it's been built by 'scraping' data from websites.

I've compared it to the latest real database (received yesterday by coincidence) that powers my venuefinder.com website and there are lots of differences.

Oddly, the real database seems to contain fewer postcodes - I have 1,649,713 rows in mine. But I know for example, I do not have PO Box postcodes which do not have coordinates anyway.

To the people talking about making this into a POI file, do you realise the file is larger than the amount of RAM in any current SatNav device? You cannot do this. It can only be used if it's compressed into a properly indexed database. It's no use to you as a POI text file.

Jellyroll · Lifetime Member Joined: May 17, 2006 Posts: 57

No problems importing into a database (of course), but 1.8 million records will overflow most spreadsheets.

If anyone wants, I can try to get it into something generic (dBase II, perhaps) that can be queried using derived (e.g. first part of Post Code, alpha portion of part of Post Code, alpha portion of OS GridRef) and existing (e.g. indexed fields such as PZ_County [37 codes incl. blank], PZ_District [220 codes incl. blank], PZ_NHS_Code [37 codes incl. blank] and _Region [11 codes incl. blank], PZ_Country [England, Northern Ireland, Scotland, Wales, Isle of Man, Channel Islands, but many blanks too]) fields. That would allow people who want to use spreadsheets to get the data via queries, and allow those using database applications other than M$ Access to get at the data without difficulty.

If I do that, can anyone (including the PGPSW site itself, subject to any legal restrictions) host the resulting file(s)?

Some fields probably contain errors. There is a St Neots post code, "PE19 6WF", that has the same PZ_County code as Stevenage, Milton Keynes and Luton post codes - it was still in Cambridgeshire last time I looked - and over half of the entries have a county code that is blank or "00".

As for whether this is of use for POI generation, I'd say that it might help those whose devices don't include post code navigation, generating a subset POI for local post codes only, for example.

pcaouolte · Posted: Thu Sep 17, 2009 12:49 pm Post subject:

NickG · Posted: Thu Sep 17, 2009 2:45 pm Post subject:

A TomTom (for example) can only handle a couple of thousand rows in a file before it becomes dog-slow. This database has over 1.5 MILLION rows. There is nothing useful you could do with that. It won't even be able to load it as a TomTom only has about 4MB of free memory when it's running, so you couldn't do any searches on it. I don't really see how it would be useful unless it was made into an indexed database you could search on. You can't search the raw text file with no index on it (even my 4GB Intel Quad core PC takes a few seconds do that!) and that has disk caching - which a sat-nav does not. Still, I'd be interested to know if it's at all usable, just out of curiosity.

pcaouolte · Posted: Thu Sep 17, 2009 7:27 pm Post subject:

navver · Posted: Thu Sep 17, 2009 8:51 pm Post subject:

my old navman POI database I bought on ebay for about 2 quid was split something like
southwest 1 southwest 2
wales1 wales2
midlands1 midlands2 midlands3
london 1 london 2 london 3
anglia 1 anglia 2
North West 1 North west 2
Scotland 1 Scotland 2 Scotland 3
etc

They were .txt or .asc files (both the same) of about 1 or 2 or 3 MB each. Navman compiled these and downloaded them to the device, each taking about an hour or more. I manged to get about 40% of the country on the card but Navman had a limit of about 24 POI files in total, but I wanted to keep space for speed zoned camera files.

The system did work well for a device that only had 5 digit postcodes built in.
_________________
Tomtom Go520, App 8.010, Map UK&ROI 810.1870

navver · Posted: Thu Sep 17, 2009 8:56 pm Post subject:

You can still buy the post code POIs on Ebay for £3.49

http://tiny.cc/eBayLink99

edited by MaFt to shorten link
_________________
Tomtom Go520, App 8.010, Map UK&ROI 810.1870

Oldboy · Posted: Fri Sep 18, 2009 6:48 am Post subject:

I don't know why users would want to pay for them. They're still available (free) in 12 files, alphabetically listed, and take up a total of 66MB as CSV Files.

In this, and probably the E-Bay case, they are a little out of date now.
_________________
Richard

TT 910 V7.903: Europe Map v1045
TT Via 135 App 12.075: Europe Map v1140

Eddiejen · Occasional Visitor Joined: Jul 02, 2006 Posts: 33

If you want to play around with the file try CSVed it loaded the file without any problems and allows you to split and manipulate the file easier
http://csved.sjfrancke.nl/index.html

NickG · Posted: Fri Sep 18, 2009 9:59 am Post subject:

Darren · Posted: Fri Sep 18, 2009 10:23 am Post subject:

Jeepers Nick, before you click send you should think about what you are posting!

Oldboy was referring to PostCode POIs, not the Royal Mail PAF database.

With regard to this thread, you yourself said you didn't think it was the RM database that Wikileaks have posted, and the Guardian, Times, The Register and many others have posted the story so if you think our post makes any difference I'm flattered but you're wrong.

Without the names and addresses it's of little use to anyone who subscribes to the PAF anyway. I strongly object to your accusation that we condone software theft, we do not and have always taken a strong line on that issue but in this case there is no suggestion that this has been stolen from anyone.
_________________
Darren Griffin

NickG · Posted: Fri Sep 18, 2009 10:33 am Post subject:

I know what he was talking about... a 7 digit postcode database is not free!

The £3000 dataset I was talking about is also not PAF data - its purely a list of 1.7M geocoded postcodes (or postcode POIs as you call them), it's in an almost identical format to the data you've linked to from this site (on wikileaks).

If postcode coordinates were free, we wouldn't have sites like http://www.freethepostcode.org/

If you don't think it's stolen and illegal, why have you described it as a "leak" then?

Darren · Posted: Fri Sep 18, 2009 10:36 am Post subject:

NickG · Posted: Fri Sep 18, 2009 10:41 am Post subject:

You wouldn't ever use the word "leak" to describe something you can legitimately download from the original owner. You'd call it a "release" wouldn't you?

Darren · Posted: Fri Sep 18, 2009 10:43 am Post subject:

True but that doesn't mean a leaked file is stolen or illegal. Whatever, the data source is not known and this story has been widely reported across the tech press.
_________________
Darren Griffin

	PocketGPS
	Web