SDSS Utilities (pydl.pydlutils)

Introduction

This package provides functionality similar to idlutils, a general suite of tools heavily used by SDSS.

idlutils is itself divided into a number of subpackages. Below we list the subpackages and the usability of the PyDL equivalent. The readiness levels are defined as:

Obsolete

No point in implementing because the purpose of the code lapsed many years ago.

Not Applicable (NA)

No point in implementing because another built-in or numpy/scipy/astropy package completely replaces this.

None

Not (yet) implemented at all.

Rudimentary

Only a few functions are implemented.

Fair

Enough functions are implemented to be useful, but some are missing.

Good

Pretty much anything you could do with the idlutils code you can do with the equivalent here.

Subpackage

Readiness Level

Comments

2mass

None

For use with matching 2MASS catalogs to SDSS data.

astrom

None

For use with SDSS astrometric data structures. Largely superseded by WCS.

bspline

Good

Fitting B-splines to data, especially for resampling.

cooling

Good

See pydl.pydlutils.cooling.read_ds_cooling().

coord

Fair

Some functionality already provided by astropy.coordinates.

cosmography

NA

Tools for computing lookback time, angular sizes at cosmological distances, etc. Use astropy.cosmology.

dimage

None

Interface to C code used for sky subtraction.

djsphot

None

A simple aperture photometry code.

dust

None

For use with the SFD galactic dust map.

first

None

For use with matching FIRST catalogs to SDSS data.

fits

NA

Use astropy.io.fits.

healpix

NA

Interact with HEALPix data. Use healpy.

image

Rudimentary

Image manipulation functions.

json

NA

Use json or other packages.

mangle

Fair

Some work still required on polygon area calculations.

math

Fair

Generic mathematical functions. Many are implemented in numpy or scipy.

mcmc

None

But there are plenty of good Python MCMC packages out there.

mglib

Obsolete

An IDL object-oriented configuration file reader.

misc

Fair

General purpose utility functions.

mpeg

None

Wrapper for ppmtompeg, makes movies from data.

mpfit

None

Appears to be an out-of-sync copy of the “markwardt” package in the The IDL® Astronomy User’s Libary.

physics

None

Implementation of physical formulas, e.g. free-free scattering.

plot

None

Much functionality already exists in matplotlib.

psf

Obsolete

Point-spread function fitting.

rgbcolor

Good

Some functionality is duplicated in astropy.visualization, especially make_lupton_rgb().

rosat

None

For use with matching ROSAT catalogs to SDSS data.

sdss

Good

Most important functionalities are bitmasks and reading sweep files.

slatec

None

Fit B-splines using C code.

spheregroup

Good

Used for matching arbitrary RA, Dec coordinates to other arbitrary RA, Dec coordinates.

TeXtoIDL

NA

This package is for including TeX in IDL plots. Since matplotlib understands TeX natively, this is not needed.

trace

Fair

Used for fitting orthogonal functions to spectroscopic wavelength solutions.

ukidss

None

Used for matching UKIDSS catalogs to SDSS data.

wise

None

Used for matching WISE catalogs to SDSS data.

yanny

Good

Tools for manipulating SDSS parameter files.

API

pydl.pydlutils Package

This subpackage implements functions from the idlutils package.

Classes

PydlutilsException

Exceptions raised by pydl.pydlutils that don't fit into a standard exception class like ValueError.

PydlutilsUserWarning

Class for warnings issued by pydl.pydlutils.

Class Inheritance Diagram

digraph inheritancef023834049 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "AstropyUserWarning" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="The primary warning class for Astropy."]; "AstropyWarning" -> "AstropyUserWarning" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AstropyWarning" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="The base warning class from which all Astropy warnings should inherit."]; "PydlException" [URL="../api/pydl.PydlException.html#pydl.PydlException",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base class for exceptions raised in PyDL functions."]; "PydlutilsException" [URL="../api/pydl.pydlutils.PydlutilsException.html#pydl.pydlutils.PydlutilsException",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Exceptions raised by :mod:`pydl.pydlutils` that don't fit into a"]; "PydlException" -> "PydlutilsException" [arrowsize=0.5,style="setlinewidth(0.5)"]; "PydlutilsUserWarning" [URL="../api/pydl.pydlutils.PydlutilsUserWarning.html#pydl.pydlutils.PydlutilsUserWarning",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class for warnings issued by :mod:`pydl.pydlutils`."]; "AstropyUserWarning" -> "PydlutilsUserWarning" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

pydl.pydlutils.bspline Module

This module corresponds to the bspline directory in idlutils.

Functions

cholesky_band(l[, mininf])

Compute lower Cholesky decomposition of a banded matrix.

cholesky_solve(a, bb)

Solve the equation \(A x = b\) where a is a lower Cholesky-banded matrix.

iterfit(xdata, ydata[, invvar, upper, ...])

Iteratively fit a B-spline set to data, with rejection.

Classes

bspline(x[, nord, npoly, bkpt, bkspread])

B-spline class.

Class Inheritance Diagram

digraph inheritancee38b3a8b55 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "bspline" [URL="../api/pydl.pydlutils.bspline.bspline.html#pydl.pydlutils.bspline.bspline",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="B-spline class."]; }

pydl.pydlutils.cooling Module

This module corresponds to the cooling directory in idlutils.

Functions

read_ds_cooling(fname[, logT])

Read in the Sutherland & Dopita (1993) cooling function.

pydl.pydlutils.coord Module

This module corresponds to the coord directory in idlutils.

Functions

current_mjd()

Return the current MJD.

munu_to_radec(munu, icrs_frame)

Convert from SDSS great circle coordinates to equatorial coordinates.

radec_to_munu(icrs_frame, munu)

Convert from equatorial coordinates to SDSS great circle coordinates.

stripe_to_eta(stripe)

Convert from SDSS great circle coordinates to equatorial coordinates.

stripe_to_incl(stripe)

Convert from SDSS stripe number to an inclination relative to the equator.

Classes

SDSSMuNu(*args[, copy, representation_type, ...])

SDSS Great Circle Coordinates

Class Inheritance Diagram

digraph inheritancef282450f08 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "BaseCoordinateFrame" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="The base class for coordinate frames."]; "ShapedLikeNDArray" -> "BaseCoordinateFrame" [arrowsize=0.5,style="setlinewidth(0.5)"]; "NDArrayShapeMethods" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Mixin class to provide shape-changing methods."]; "SDSSMuNu" [URL="../api/pydl.pydlutils.coord.SDSSMuNu.html#pydl.pydlutils.coord.SDSSMuNu",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="SDSS Great Circle Coordinates"]; "BaseCoordinateFrame" -> "SDSSMuNu" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ShapedLikeNDArray" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Mixin class to provide shape-changing methods."]; "NDArrayShapeMethods" -> "ShapedLikeNDArray" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

pydl.pydlutils.image Module

This module corresponds to the image directory in idlutils.

Functions

djs_maskinterp(yval, mask[, xval, axis, const])

Interpolate over masked pixels in a vector, image or 3-D array.

djs_maskinterp1(yval, mask[, xval, const])

Interpolate over a masked, 1-d array.

pydl.pydlutils.mangle Module

This module corresponds to the mangle directory in idlutils.

Mangle [1] is a software suite that supports the concept of polygons on a sphere, and is used to, for example, describe the window function of the Sloan Digital Sky Survey.

This implementation is intended to support the portions of Mangle that are included in idlutils. To simplify the coding somewhat, unlike idlutils, the caps information is accessed through polygon.x and polygon.cm, not polygon.caps.x or polygon.caps.cm.

Window function operations are already supported by is_in_polygon() and is_in_window(). However, calculation of polygon area (solid angle) from first principles is not yet implemented.

Note that in traditional geometry “spherical polygon” means a figure bounded by great circles. Mangle allows polygons to be bounded by any circle, great or not. Use care when comparing formulas in this module to formulas in the mathematical literature.

References

1

Swanson, M. E. C.; Tegmark, Max; Hamilton, Andrew J. S.; Hill, J. Colin, 2008 MNRAS 387, 1391.

Functions

angles_to_x(points[, latitude])

Convert spherical angles to unit Cartesian vectors.

cap_distance(x, cm, points)

Compute the distance from a point to a cap, and also determine whether the point is inside or outside the cap.

circle_cap(radius, points)

Construct caps based on input points (directions on the unit sphere).

is_cap_used(use_caps, i)

Returns True if a cap is used.

is_in_cap(x, cm, points)

Are the points in a (single) given cap?

is_in_polygon(polygon, points[, ncaps])

Are the points in a given (single) polygon?

is_in_window(polygons, points[, ncaps])

Check to see if points lie within a set of polygons.

read_fits_polygons(filename[, convert])

Read a "polygon" format FITS file.

read_mangle_polygons(filename)

Read a "polygon" format ASCII file in Mangle's own format.

set_use_caps(polygon, index_list[, add, ...])

Set the bits in use_caps for a polygon.

x_to_angles(points[, latitude])

Convert unit Cartesian vectors to spherical angles.

Classes

FITS_polygon(input)

Handle polygons read in from a FITS file.

ManglePolygon(*args, **kwargs)

Simple object to represent a polygon.

PolygonList(*args, **kwargs)

A list that contains ManglePolygon objects and possibly some metadata.

Class Inheritance Diagram

digraph inheritancee89c31c3af { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "FITS_polygon" [URL="../api/pydl.pydlutils.mangle.FITS_polygon.html#pydl.pydlutils.mangle.FITS_polygon",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Handle polygons read in from a FITS file."]; "FITS_rec" -> "FITS_polygon" [arrowsize=0.5,style="setlinewidth(0.5)"]; "FITS_rec" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="FITS record array class."]; "recarray" -> "FITS_rec" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ManglePolygon" [URL="../api/pydl.pydlutils.mangle.ManglePolygon.html#pydl.pydlutils.mangle.ManglePolygon",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Simple object to represent a polygon."]; "PolygonList" [URL="../api/pydl.pydlutils.mangle.PolygonList.html#pydl.pydlutils.mangle.PolygonList",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A :class:`list` that contains :class:`ManglePolygon` objects and"]; "ndarray" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="ndarray(shape, dtype=float, buffer=None, offset=0,"]; "recarray" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Construct an ndarray that allows field access using attributes."]; "ndarray" -> "recarray" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

pydl.pydlutils.math Module

This module corresponds to the math directory in idlutils.

Functions

djs_median(array[, dimension, width, boundary])

Compute the median of an array.

djs_reject(data, model[, outmask, inmask, ...])

Routine to reject points when doing an iterative fit to data.

find_contiguous(x)

Find the longest sequence of contiguous non-zero array elements.

Classes

computechi2(bvec, sqivar, amatrix)

Solve the linear set of equations \(A x = b\) using SVD.

Class Inheritance Diagram

digraph inheritance567431bde1 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "computechi2" [URL="../api/pydl.pydlutils.math.computechi2.html#pydl.pydlutils.math.computechi2",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Solve the linear set of equations :math:`A x = b` using SVD."]; }

pydl.pydlutils.misc Module

This module corresponds to the misc directory in idlutils.

Functions

decode_mixed(x)

Convert bytes in Numpy arrays into strings.

djs_laxisgen(dims[, iaxis])

Returns an integer array where each element of the array is set equal to its index number along the specified axis.

djs_laxisnum(dims[, iaxis])

Returns an integer array where each element of the array is set equal to its index number in the specified axis.

hogg_iau_name(ra, dec[, prefix, precision])

Properly format astronomical source names to the IAU convention.

hogg_iau_name_main()

struct_print(array[, filename, formatcodes, ...])

Print a NumPy record array (analogous to an IDL structure) in a nice way.

pydl.pydlutils.rgbcolor Module

This module corresponds to the rgbcolor directory in idlutils.

This code reproduces the algorithms of Lupton et al. (2004) [2]. The purpose is to produce nice color (RGB, JPEG, PNG, etc.) from FITS images in astronomical filter bands.

References

2

Lupton, Robert, et al., 2004 PASP 116, 113.

Functions

nw_arcsinh(colors[, nonlinearity])

Scales colors by a degree of nonlinearity specified by user.

nw_cut_to_box(colors[, origin])

Limits the pixel values of the image to a 'box', so that the colors do not saturate to white but to a specific color.

nw_float_to_byte(image[, bits])

Converts an array of floats in [0.0, 1.0] to bytes in [0, 255].

nw_scale_rgb(colors[, scales])

Multiply RGB image by color-dependent scale factor.

pydl.pydlutils.sdss Module

This module corresponds to the sdss directory in idlutils.

Functions

default_skyversion()

Returns skyversion number to use for photoop imaging.

sdss_astrombad(run, camcol, field[, ...])

For a list of RUN, CAMCOL, FIELD, return whether each field has bad astrometry.

sdss_flagexist(flagname, bitname[, ...])

Check for the existence of flags.

sdss_flagname(flagname, flagvalue[, concat])

Return a list of flag names corresponding to the values.

sdss_flagval(flagname, bitname)

Convert bitmask names into values.

sdss_objid(run, camcol, field, objnum[, ...])

Convert SDSS photometric identifiers into CAS-style ObjID.

sdss_specobjid(plate, fiber, mjd, run2d[, ...])

Convert SDSS spectrum identifiers into CAS-style specObjID.

sdss_sweep_circle(ra, dec, radius[, stype, ...])

Read the SDSS datasweep files and return objects around a location.

set_maskbits([idlutils_version, maskbits_file])

Populate the maskbits cache.

unwrap_specobjid(specObjID[, run2d_integer, ...])

Unwrap CAS-style specObjID into plate, fiber, mjd, run2d.

pydl.pydlutils.spheregroup Module

This module corresponds to the spheregroup directory in idlutils.

Functions

spheregroup(ra, dec, linklength[, chunksize])

Perform friends-of-friends grouping given ra/dec coordinates.

spherematch(ra1, dec1, ra2, dec2, matchlength)

Match points on a sphere.

Classes

chunks(ra, dec, minSize)

chunks class

groups(coordinates, distance[, separation])

Group a set of objects (a list of coordinates in some space) based on a friends-of-friends algorithm

Class Inheritance Diagram

digraph inheritance42f0133a1b { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "chunks" [URL="../api/pydl.pydlutils.spheregroup.chunks.html#pydl.pydlutils.spheregroup.chunks",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="chunks class"]; "groups" [URL="../api/pydl.pydlutils.spheregroup.groups.html#pydl.pydlutils.spheregroup.groups",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Group a set of objects (a list of coordinates in some space) based on"]; }

pydl.pydlutils.trace Module

This module corresponds to the trace directory in idlutils.

Traceset (or “trace set”):

A set of coefficient vectors defining a set of functions over a common independent-variable domain specified by xmin and xmax values. The functions in the set are defined in terms of a linear combination of basis functions (such as Legendre of Chebyshev polynonials) up to a specified maximum order, weighted by the values in the coefficient vectors, and evaluated using a suitable affine rescaling of the 1dependent-variable domain (such as [xmin, xmax] -> [-1, 1] for Legendre polynomials). For evaluation purposes, the domain is assumed by default to be a zero-based integer baseline from xmin to xmax such as would correspond to a digital detector pixel grid.

Etymology:

From the original motivating use case of encoding the “traces” of multiple spectra across the detector in a multi-fiber spectrograph.

Definition from A. Bolton, U. of Utah, June 2015.

Functions

fchebyshev(x, m)

Compute the first m Chebyshev polynomials.

fchebyshev_split(x, m)

Compute the first m Chebyshev polynomials, but modified to allow a split in the baseline at \(x=0\).

fpoly(x, m)

Compute the first m simple polynomials.

func_fit(x, y, ncoeff[, invvar, ...])

Fit x, y positions to a functional form.

traceset2xy(tset[, xpos, ignore_jump])

Convert from a trace set to an array of x,y positions.

xy2traceset(xpos, ypos, **kwargs)

Convert from x,y positions to a trace set.

Classes

TraceSet(*args, **kwargs)

Implements the idea of a trace set.

Class Inheritance Diagram

digraph inheritance5a4625f7af { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "TraceSet" [URL="../api/pydl.pydlutils.trace.TraceSet.html#pydl.pydlutils.trace.TraceSet",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Implements the idea of a trace set."]; }

pydl.pydlutils.yanny Module

This module corresponds to the yanny directory in idlutils.

This is a Python library for reading & writing yanny files.

yanny is an object-oriented interface to SDSS Parameter files (a.k.a. “FTCL” or “yanny” files) following these specifications. Parameter files typically have and can be recognized by the extension .par. These files may also be recognized if the first line of the file is:

#%yanny

This is not part of the standard specification, but it suffices to identify, e.g., files that have not yet been written to disk, but only exist as file objects.

Because Parameter files can contain multiple tables, as well as metadata, there is no simple, one-to-one correspondence between these files and, say, an astropy Table object. Thus, yanny objects are implemented as a subclass of OrderedDict (to remember the order of keyword-value pairs), and the actual data are values accessed by keyword. Still, it is certainly possible to write a table-like object to a yanny file.

Given the caveats above, we have introduced experimental support for reading and writing yanny files directly to/from Table objects. Because of the experimental nature of this support, it must be activated “by hand” by including this snippet in your code:

from astropy.table import Table
from astropy.io.registry import (register_identifier, register_reader,
                                 register_writer)
from pydl.pydlutils.yanny import (is_yanny, read_table_yanny,
                                  write_table_yanny)

register_identifier('yanny', Table, is_yanny)
register_reader('yanny', Table, read_table_yanny)
register_writer('yanny', Table, write_table_yanny)

Currently multidimensional arrays are only supported for type char, and a close reading of the specifications indicates that multidimensional arrays were only ever intended to be supported for type char. So no multidimensional arrays, sorry.

Functions

is_yanny(origin, path, fileobj, *args, **kwargs)

Identifies Yanny files or objects.

read_table_yanny(filename[, tablename])

Read a yanny file into a Table.

write_ndarray_to_yanny(filename, datatables)

Converts a NumPy record array into a new FTCL/yanny file.

write_table_yanny(table, filename[, ...])

Write a Table to a yanny file.

Classes

yanny([filename, raw])

An object interface to a yanny file.

Class Inheritance Diagram

digraph inheritance4ee99d8f13 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "OrderedDict" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Dictionary that remembers insertion order"]; "yanny" [URL="../api/pydl.pydlutils.yanny.yanny.html#pydl.pydlutils.yanny.yanny",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An object interface to a yanny file."]; "OrderedDict" -> "yanny" [arrowsize=0.5,style="setlinewidth(0.5)"]; }