The All-Sky Portable (ASP) optical catalogue Eric Wim Flesch, 14 May 2017 ================================================================================ This "All-Sky Portable" (ASP) catalogue has 1,163,237,190 optical sources stored at 10 bytes per source. It is presented as 15 zipped files of 9Gb total size which unzip into 100 ASP files of 11Gb total size. The catalog can be downloaded from the PASA datastore at http://dx.doi.org/10.4225/50/5807fbc12595f . Data are from the APM (http://www.ast.cam.ac.uk/~mike/apmcat), USNO-B1.0 and USNO-A1.0 (http://www.nofs.navy.mil), and SDSS (http://sdss.org) catalogues. Each 10-byte record presents J2000 astrometry to 1/10th arcsecond precision, red-blue magnitudes to 1/100th magnitude precision with PSFs and variability indicator, and flags for proper motion, epoch and, for each of photometry and astrometry, identification of the source survey & catalog. The SDSS data's accuracy matches this precision, and for other data the astrometry is usually (81.6%) accurate to within 0.5 arcsecond, athough accuracy lessens to 2-3 arcseconds for data near edges of Schmidt photographic plates. Photometry is recalibrated for APM and USNO-A/B data (see Note 4 below) and is usually accurate to 0.3 magnitude, at least internally. Notes: (1) Some BASIC & Python code to read these files is at bottom. (2) All data was de-duplicated but some elusive duplicates still lurk. (3) 25M epoch-1955 data are presented alongside other data; they are flagged. (4) Fuzzy (galaxy) PSFs from APM/USNO-B usually display too-bright magnitudes because their magnitudes were calculated using a stellar PSF model. (5) Spurious sources can appear in the star spikes of bright stars. (6) Dense data from high-density places like the Galactic Bulge or Magellanic clouds are interwoven with spurious data caused by sky value saturation, so use with considerable caution there. Questions/comments/praise/complaints may be directed to me at eric@flesch.org. Author's home page for this catalog is http://quasars.org/asp.html . File Summary: -------------------------------------------------------------------------------- FileName Lrecl Records Explanations -------------------------------------------------------------------------------- N00-N49, S00-S49 (.asp) 10 1163237190 The catalogue (100 files) (1) ASP-ReadMe.txt 80 . This file -------------------------------------------------------------------------------- Byte-by-byte Description of the ASP (all-sky portable) file: -------------------------------------------------------------------------------- Bytes Format Units Label Explanations -------------------------------------------------------------------------------- 1 A1 RAhour/10 RAhour 1/10th of RA hour, ascii range 0-239 (2) 2-3 PIB2 arcsec/10 RAoff right ascension offset (3) 4-5 PIB2 arcsec/10 DECoff declination offset (3) 6-7 IB2 (mag/100) Rphot red optical magnitude, PSF, variability (4) 8-9 IB2 (mag/100) Bphot blue optical magnitude, PSF, variability (4) 10 A1 --- Flags prop motion, epoch, source survey & catalog (5) -------------------------------------------------------------------------------- Note (1): ASP consists of 100 files, with file names of N00, N01 ... N49.asp for Northern hemisphere files, and S00, S01 ... S49.asp for Southern files. Each file is a sky circle containing all RA for a DEC 1.8 degree wide band, so the 100 files (sky circles) stack from pole to pole. The file N00.asp is for DEC 0 (inclusive) to +1.8 (exclusive) degrees, going upwards to file N49.asp which has DEC +88.2 to +90 degrees. The Southern files mirror the Northern, with S00.asp at the equator down to S49.asp at the pole. The file name gives its DEC baseline stated as a multiple of 1.8 degrees, e.g., the file "N17.asp" has the DEC baseline of 17 x 1.8 = 30.6 degrees. The offset given by the DECoff field (written in units of tenth arcsecond) is added to the DEC baseline to give the J2000 declination of its object. Note (2): This gives the RA baseline onto which to add the RAoff to get the RA. It is a 1-byte field, the ascii value of which is the datum expressed in tenths of a RA hour, e.g., "115" = 11.5 RA hours = 172.5 RA degrees = 621000.0 RA arcseconds. The offset given by the RAoff field (written in units of tenth arcsecond) is added to this RA baseline to give the J2000 right ascension of the source. Note (3): RAoff and DECoff are offsets from their respective RA & DEC baselines (see (1) and (2), above). They are in units of 1/10th of an arcsecond, which for RA means 1/36000th of a RA degree. RAoff has a range of 0 to 53999, and DECoff has a range of 0 to 64799. These 2-digit numbers are in positive (i.e., unsigned) integer binary little-endian format, which is the default unsigned format for PCs. Most programming languages provide this as a data type, e.g., "UShort" in BASIC, but if that is not available in your programming language, improvise thusly: use a 2-byte "integer" type in reading RAoff & DECoff, and accept any ensuing positive number, but for any negative number add 65536 which gives the correct integer value in the range 32768-65535. The J2000 coordinates of each object are RA = RA baseline + RA offset and DEC = DEC baseline + DEC offset. As an example, take record #3855424 of the file "N33.asp", with data of RAhour=107, RAoff=37206, DECoff=19490. For this record, the RA baseline is (RAhour/10) x (54000 arcsec / RA hour) = 577800 RA arcseconds, and the DEC baseline (using the file name #) is 33 x 1.8 (degrees / file#) x 3600 (arcsecs / degree) = 213840 arcseconds. Adding the offsets to the baselines, RA = 577800 + 37206/10 = 581520.6 RA arcsecs, and DEC = 213840 + 19490/10 = 215789.0 arcsecs. Divide this (581520.6, 215789.0) by 3600 to get (161.5335, 59.9414) in J2000 degrees. Compute southern hemisphere files exactly the same as northern files, treating all declinations and offsets, including the baseline, as positive. Then, as your last act, affix the "-" to make the declination negative. Note (4): These are 2-digit integers in little-endian format, common on PCs. These are red and blue photometry and usually show a 3-to-5 digit number, but can be zero which means that there is no data (no signal or confused) for that band. Non-zero numbers like "20143" break down as: (a) The number excluding the last digit, e.g., "2014", which is the magnitude in hundredths, thus 2014 --> 20.14. (b) The last digit gives the PSF and notional variability of that band. Legend: 0,1,2 = unknown PSF 3,4,5 = point source, unresolved, stellar. 6,7,8 = fuzzy source, resolved, e.g., galaxy. 9 = inferred* source, rare, no PSF. Within each set of 3 digits (e.g., 3,4,5 for point sources), the first digit shows no variability encountered during processing (the object can be seen repeatedly on overlapping plates and surveys, so the magnitudes are compared then), the 2nd digit shows variability of one magnitude seen, and the 3rd digit shows variability of 2+ magnitudes seen. Thus for point sources: 3 = point source, no variability seen 4 = point source, variability of 1 magnitude seen 5 = point source, variability of 2+ magnitudes seen Such flagged variability is indicative only, because variations can also arise due to object confusion, sky value offsets, or plate artefacts, so any flagged variability needs individual confirmation. APM magnitudes have been recalibrated using shared stars on plate margins. USNO-A/B magnitudes have been normalized onto APM data for shared plates, and bulk-adjusted elsewhere using the average normalization. APM/USNO-B magnitudes of fuzzy sources are usually shown too bright because those magnitudes were calculated using a stellar PSF model. See the ASP paper for details of these. * Inferred sources: this object was originally reported in a single band only, but the other band has been "inferred" by combining an APM ellipse with a USNO-A2.0 point source positioned at one end of that ellipse, thus allowing the object at the other end to be astrometrically calculated -- and which matches to this single-band object. Inferred magnitudes have an expected error of 1-2 magnitudes, but its primary deliverable is that the object is seen in both bands. Note (5): This 1-byte character has 8 bit flags; 1 = flag set, 0 = not set. The flags are: bits 1-3: Photometry is from the following survey & catalog: 000: POSS-I from the APM, blue is O 4050A. 001: POSS-I from the USNO-A, blue is O 4050A. 010: POSS-I from the USNO-B, blue is O 4050A. 011: UKST from the APM, blue is Bj 4850A. 100: UKST from the USNO-B, unless the epoch 1955 flag is set, in which case USNO-A is the provenance. Blue is Bj 4850A. 101: POSS-II from the USNO-B, blue is Bj 4850A. 110: Tycho-2 stars from the USNO-B, blue is Tycho BT 4400A. 111: SDSS data from the SDSS DR9 datasweeps catalog, unless the epoch 1955 flag is set, in which case this is SDSS DR8 data from the XDQSO catalog of Bovy et al 2011,ApJ,729,141. bits 4-6: Astrometry flag which is the same as bits 1-3. One difference: 110: Epoch 2000 position as presented by USNO-B. This is shown to prevent user confusion, but is not complete. See the "epoch" flag, below, for the complete flag. bits 7-8: Epoch flag and proper motion indicator. 01: Epoch 1955 (E1955) flag for POSS-I data, showing that this object was unmatched to any later survey. Other uses: If UKST data has this flag, it denotes USNO-A provenance. If SDSS data has this flag, it denotes XDQSO provenance, i.e., from the catalog of Bovy et al 2011,ApJ,729,141. 10: Epoch 2000 (E2000) as presented by the USNO-B, which projected this for multi-epoch objects. This is complete except that proper motion (below) also usually shows E2000. 11: Proper Motion object as flagged by USNO-B as having a 90% probability of being true. However, the USNO-B greatly over-reported (50x) the incidence of proper motion, for completeness. Astrometry is reported at the USNO-B E2000 position, unless SDSS astrometry (E2005) is reported. 00: Object reported at the epoch of the astrometry survey. Those epochs are: POSS-I: observed 1949-1958, mean epoch 1955 UKST: observed 1978-1990, mean epoch 1985 POSS-II: observed 1985-1999, mean epoch 1993 SDSS: observed 2000-2009, mean epoch 2005 -------------------------------------------------------------------------------- Code to read ASP files follows, first some BASIC code, then Python code. -------------------------------------------------------------------------------- (1) Some BASIC code to read these files: ASPfile$ = "N01" '<--- specify file to read   Type ASProw     'use UShort (positive integers) for RAoff & DECoff if available     RAhour As Byte: RAoff As Integer: DECoff As Integer: _ Rphot As Integer: Bphot As Integer: flags As Byte End Type Dim ASPn As ASProw   Open "c:\ASP\" + ASPfile$ + ".asp" For Random As #1 Len = 10 'assumes "c:\ASP"   Number_records = LOF(1) \ 10   Hemi = UCase$(Mid$(ASPfile$, 1, 1)) DEC0 = 64800 * Mid$(ASPfile$, 2, 2)    ' declination baseline   For i = 1 To Number_records     Get 1, i, ASPn     RA0 = 54000 * ASPn.RAhour     ' RA baseline     RAoff = ASPn.RAoff     If RAoff < 0 Then RAoff = RAoff + 65536      'if not using positive integers     DECoff = ASPn.DECoff      If DECoff < 0 Then DECoff = DECoff + 65536   'if not using positive integers     RAdeg = (RA0 + RAoff) / 36000     DECdeg = (DEC0 + DECoff) / 36000: If Hemi = "S" Then DECdeg = -DECdeg       Rphot = ASPn.Rphot     If Rphot > 0 Then         Rmag = (Rphot \ 10) / 100 ' "\" is the "int" function (truncates)         Psf = Mod(Rphot, 10) ' or: Psf = Rphot - 10 * (Rphot \ 10)         If Psf <= 2 Then            Rpsf = "none "         ElseIf Psf <= 5 Then Rpsf = "star "         ElseIf Psf <= 8 Then Rpsf = "fuzz "         Else: Rpsf = "infer"         End If Rvar = Mod(Psf, 3) ' variable -- or: Rvar = Psf - 3 * (Psf \ 3)     Else: Rmag = 0: Rpsf = " "     End If       Bphot = ASPn.Bphot    If Bphot > 0 Then         Bmag = (Bphot \ 10) / 100 ' "\" is the "int" function (truncates)         Psf = Mod(Bphot, 10) ' or: Psf = Bphot - 10 * (Bphot \ 10)         If Psf <= 2 Then             Bpsf = "none "         ElseIf Psf <= 5 Then Bpsf = "star "         ElseIf Psf <= 8 Then Bpsf = "fuzz "         Else: Bpsf = "infer"         End If Bvar = Mod(Psf, 3) ' variable -- or: Bvar = Psf - 3 * (Psf \ 3)     Else: Bmag = 0: Bpsf = " "     End If   flagn = ASPn.flags ' get Epoch first because 15 lines down it is used as a flag     If flagn >= 128 Then Epoch = 2000     ElseIf flagn >= 64 Then Epoch = 1955 Else: Epoch = 0     End If ' photometry flags signal the source survey/catalog of the magnitudes photom = Mod(flagn, 8) ' or: photom = flagn - 8 * (flagn \ 8) If photom = 1 Then survey = "POSS-I": catalog = "USNO-A" ElseIf photom = 2 Then survey = "POSS-I": catalog = "USNO-B" ElseIf photom = 3 Then survey = "UKST ": catalog = "APM " ElseIf photom = 4 Then survey = "UKST ": catalog = "USNO-B" If Epoch = 1955 Then catalog = "USNO-A": Epoch = 0 ElseIf photom = 5 Then survey = "POSS-2": catalog = "USNO-B" ElseIf photom = 6 Then survey = "Tycho2": catalog = "USNO-B" ElseIf photom = 7 Then survey = "SDSS ": catalog = "SDSS " If Epoch = 1955 Then catalog = "XDQSO ": Epoch = 0 Else: survey = "POSS-I": catalog = "APM " 'photom=0 shows APM POSS-I End If ' astrometry flags signal the source survey/catalog of the RA/DEC astrom = Mod(flagn, 64) \ 8 ' "\" is the "int" function (truncates) ' or: astrom = (flagn-64*(flagn\64))\8 If astrom = 1 Then surveya = "POSS-I": cataloga = "USNO-A" ElseIf astrom = 2 Then surveya = "POSS-I": cataloga = "USNO-B" ElseIf astrom = 3 Then surveya = "UKST ": cataloga = "APM " ElseIf astrom = 4 Then surveya = "UKST ": cataloga = "USNO-B" If survey = "UKST " And catalog = "USNO-A" Then cataloga = "USNO-A" ElseIf astrom = 5 Then surveya = "POSS-2": cataloga = "USNO-B" ElseIf astrom = 6 Then surveya = "Ep2000": cataloga = "USNO-B" ElseIf astrom = 7 Then surveya = "SDSS ": cataloga = "SDSS " If catalog = "XDQSO " Then cataloga = "XDQSO " Epoch = 2005 Else: surveya = "POSS-I": cataloga = "APM " End If    If flagn >= 192 Then proper_motion = "Y" If Epoch = 0 Then If surveya = "UKST " Then Epoch = 1985 If surveya = "POSS-2" Then Epoch = 1993     End If [Keep ASPfile$ i RAdeg DECdeg Rmag Rpsf Rvar Bmag Bpsf Bvar survey catalog surveya cataloga proper_motion Epoch] [OUTPUT to your destination table here] Next Close #1 -------------------------------------------------------------------------------- The above code delivers these values for the 1st 20 lines of the file "N01.asp": survey catalog survey prop RAdeg DECdeg Rmag Rpsf Bmag Bpsf (phot) (phot) (astr) mot epoch Rvar ------- ------- ----- ---- ----- ---- ------ ------ ------ --- ----- ---- 0.00008 1.84436 19.55 fuzz 22.01 star UKST APM SDSS 2005 0.00008 2.30656 22.19 fuzz 23.29 fuzz SDSS SDSS SDSS 2005 0.00011 3.35200 17.14 star 18.27 star POSS-I APM SDSS Y 2005 0.00014 2.31425 18.52 star 21.53 star POSS-I APM SDSS 2005 0.00017 1.88694 15.34 star 17.05 star POSS-I APM SDSS Y 2005 0.00019 3.36803 0.00 none 21.38 fuzz POSS-I USNO-B POSS-2 2000 0.00019 3.58125 19.93 fuzz 20.98 fuzz POSS-2 USNO-B SDSS 2005 0.00022 2.46600 20.08 fuzz 21.47 none POSS-I USNO-B POSS-I 1955 0.00025 2.29492 22.10 fuzz 22.74 fuzz SDSS SDSS SDSS 2005 0.00028 2.26714 19.99 star 21.25 star UKST APM SDSS 2005 0.00031 3.12378 21.85 fuzz 22.48 fuzz SDSS SDSS SDSS 2005 0.00036 1.91664 22.37 fuzz 22.91 fuzz SDSS SDSS SDSS 2005 0.00036 3.57567 23.52 star 25.10 star SDSS SDSS SDSS 2005 0.00047 3.54000 22.02 fuzz 22.58 fuzz SDSS SDSS SDSS 2005 0.00056 2.36894 18.39 star 20.59 star POSS-I APM SDSS 2005 0.00058 3.55719 21.85 fuzz 22.53 fuzz SDSS SDSS SDSS 2005 0.00075 2.11856 21.82 star 23.25 star SDSS SDSS SDSS 2005 0.00075 2.29625 22.40 fuzz 23.06 fuzz SDSS SDSS SDSS 2005 0.00075 3.10669 18.01 star 20.15 star POSS-I APM SDSS Y 2005 0.00075 3.43792 19.77 fuzz 20.93 star POSS-2 USNO-B SDSS 2005 1 -------------------------------------------------------------------------------- (2) Python code to read these files: # this PYTHON program reads and displays an all-sky portable (ASP) file. # data is displayed as written in file. # author: Reggis Sanders # date of initial creation: 10 January 2017 # updated 29 March 2017 file_name = "N01.asp" num_records = 120 added_declination = 64800 * int(file_name[1:3]) # 64800 = 10 * 3600 (arcsecs in degree) * 1.8 negative_declination = file_name[:1] == "S" psf_chart = "unknown ", "stellar ", "fuzzy ", "inferred" variability_chart = " ", "1 ", "2+" def get_flags(flag_byte, red_var): change_red_var = False epoch_pm = flag_byte // 64 if epoch_pm == 1: epoch = "1955" elif epoch_pm >= 2: epoch = "2000" else: epoch = "" current_bits = flag_byte % 8 if current_bits == 1: ph_survey = "POSS-I " ph_catalog = "USNO-A" elif current_bits == 2: ph_survey = "POSS-I " ph_catalog = "USNO-B" elif current_bits == 3: ph_survey = "UKST " ph_catalog = "APM " elif current_bits == 4: ph_survey = "UKST " if epoch == "1955": ph_catalog = "USNO-A" epoch = "" else: ph_catalog = "USNO-B" elif current_bits == 5: ph_survey = "POSS-II" ph_catalog = "USNO-B" elif current_bits == 6: ph_survey = "Tycho-2" ph_catalog = "USNO-B" elif current_bits == 7: ph_survey = "SDSS " if epoch == "1955": ph_catalog = "XDQSO " epoch = "" else: ph_catalog = "SDSS " else: ph_survey = "POSS-I " ph_catalog = "APM " current_bits = (flag_byte % 64) // 8 if current_bits == 1: as_survey = "POSS-I " as_catalog = "USNO-A" elif current_bits == 2: as_survey = "POSS-I " as_catalog = "USNO-B" elif current_bits == 3: as_survey = "UKST " as_catalog = "APM " elif current_bits == 4: as_survey = "UKST " if ph_survey == "UKST " and ph_catalog == "USNO-A": as_catalog = "USNO-A" else: as_catalog = "USNO-B" elif current_bits == 5: as_survey = "POSS-II" as_catalog = "USNO-B" elif current_bits == 6: as_survey = "Ep2000 " as_catalog = "USNO-B" elif current_bits == 7: as_survey = "SDSS " if ph_catalog == "XDQSO ": as_catalog = "XDQSO " else: as_catalog = "SDSS " epoch = "2005" else: as_survey = "POSS-I " as_catalog = "APM " if epoch_pm == 3: proper_motion = "prop mot" else: proper_motion = " " if epoch == "": if as_survey == "UKST ": epoch = "1985" elif as_survey == "POSS-II": epoch = "1993" flags = (ph_survey + " " + ph_catalog + " " + as_survey + " " + as_catalog + " " + proper_motion + " " + epoch) return (flags, change_red_var) def display_records(num): record_bytes = open(file_name, mode="rb").read()[:10 * num] for i in range(num): added_ra = record_bytes[10 * i] * 54000 ra_offset = int.from_bytes(record_bytes[10 * i + 1:10 * i + 3], byteorder="little") declination = int.from_bytes(record_bytes[10 * i + 3:10 * i + 5], byteorder="little") red_mag_psf = int.from_bytes(record_bytes[10 * i + 5:10 * i + 7], byteorder="little") blue_mag_psf = int.from_bytes(record_bytes[10 * i + 7:10 * i + 9], byteorder="little") declination += added_declination right_ascension = added_ra + ra_offset if negative_declination: declination = -declination right_ascension = format(right_ascension / 36000, ".5f").zfill(9) declination = format(declination / 36000, "+.5f").zfill(9) if red_mag_psf == 0: red_brightness = " " red_psf = "no red " else: red_brightness = format(red_mag_psf // 10 / 100, ".2f").zfill(5) red_psf = psf_chart[red_mag_psf % 10 // 3] red_variability = variability_chart[(red_mag_psf % 10) % 3] if blue_mag_psf == 0: blue_brightness = " " blue_psf = "no blue " else: blue_brightness = format(blue_mag_psf // 10 / 100, ".2f").zfill(5) blue_psf = psf_chart[blue_mag_psf % 10 // 3] blue_variability = variability_chart[(blue_mag_psf % 10) % 3] flags = get_flags(record_bytes[10 * i + 9], red_variability) flag_string = flags[0] if flags[1]: red_variability = " " print(right_ascension, declination, red_brightness, red_psf, red_variability, blue_brightness, blue_psf, blue_variability, flag_string) # main routine if __name__ == "__main__": display_records(num_records) input("") The above code delivers these values for the 1st 20 lines of the file "N01.asp": survey catalog survey prop RAdeg DECdeg Rmag Rpsf Rvar Bmag Bpsf Bvar (phot) (phot) (astr) mot --------- --------- ----- ------- -- ----- ------- -- ------- ------ ------- --- 000.00008 +01.84436 19.55 fuzzy 22.01 stellar UKST APM SDSS 000.00008 +02.30656 22.19 fuzzy 23.29 fuzzy SDSS SDSS SDSS 000.00011 +03.35200 17.14 stellar 18.27 stellar POSS-I APM SDSS pro 000.00014 +02.31425 18.52 stellar 21.53 stellar POSS-I APM SDSS 000.00017 +01.88694 15.34 stellar 17.05 stellar POSS-I APM SDSS pro 000.00019 +03.36803 no red 21.38 fuzzy POSS-I USNO-B POSS-II 000.00019 +03.58125 19.93 fuzzy 20.98 fuzzy POSS-II USNO-B SDSS 000.00022 +02.46600 20.08 fuzzy 21.47 unknown POSS-I USNO-B POSS-I 000.00025 +02.29492 22.10 fuzzy 22.74 fuzzy SDSS SDSS SDSS 000.00028 +02.26714 19.99 stellar 21.25 stellar UKST APM SDSS 000.00031 +03.12378 21.85 fuzzy 22.48 fuzzy SDSS SDSS SDSS 000.00036 +01.91664 22.37 fuzzy 22.91 fuzzy SDSS SDSS SDSS 000.00036 +03.57567 23.52 stellar 25.10 stellar SDSS SDSS SDSS 000.00047 +03.54000 22.02 fuzzy 22.58 fuzzy SDSS SDSS SDSS 000.00056 +02.36894 18.39 stellar 20.59 stellar POSS-I APM SDSS 000.00058 +03.55719 21.85 fuzzy 22.53 fuzzy SDSS SDSS SDSS 000.00075 +02.11856 21.82 stellar 23.25 stellar SDSS SDSS SDSS 000.00075 +02.29625 22.40 fuzzy 23.06 fuzzy SDSS SDSS SDSS 000.00075 +03.10669 18.01 stellar 20.15 stellar POSS-I APM SDSS pro 000.00075 +03.43792 19.77 fuzzy 1 20.93 stellar 1 POSS-II USNO-B SDSS ----------------------- end of Python program & ReadMe -----------------------