What happens to an image of the sky after it’s taken by the Dark Energy Camera? This is where the Dark Energy Survey Data Management (DESDM) team comes in, providing the infrastructure needed to store the data and maximize its scientific yield. After an image leaves the Chilean observatory, it is sent to the National Center for Supercomputing Applications (NCSA) at the University of Illinois in Urbana-Champaign, IL (UIUC). Once there, the DESDM team “processes” each image. The DESDM team provides all the infrastructure to process, manage, and store the images and scientific data. DESDM is comprised of astronomers, computer scientists, and system engineers who work together at the intersection of scientific research and the computing systems that enable it.
Each DECam image is a gigabyte in size. The Dark Energy Survey will take about 400 of these extremely large images per night. This presents a very high data-collection rate for an astronomy experiment. DESDM processes the images on large computing clusters at NCSA and at Fermilab. Many images can be processed simultaneously, each taking a few hours. A full night’s worth of images (about 300) can produce as much as 2.5 terabytes of data. That means each night DESDM produces enough data to fill about 4 laptop computers!
DESDM has a critical role in making sure the images taken by the Dark Energy Camera meet the requirements necessary for the Dark Energy Survey (DES) to achieve its science goals. DESDM performs a quality assessment of every image, and if an image doesn’t meet requirements, sends a notice to the telescope to retake that image. By retaking poor quality images we maximize the total amount of good data that can be used for research.
What exactly does it mean to “process” an image? For the most part, processing an image consists of removing electronic noise introduced by the camera and telescope, masking out unwanted “junk” like bright trails left by satellites and airplanes flying over the telescope while the image was taken, and making scientific measurements (e.g., brightness, position, shape) of the millions of stars and galaxies detected in each image.
DES takes images only half of the year. During the other half of the year, we update and make improvements to the DESDM system. This involves improving scientific algorithms (how scientific values are measured), fixing any bugs in the codes, upgrading the computing and storage infrastructure, running numerous tests, and validating image and science quality.
Once these tests have passed all operational and scientific requirements, the DESDM team reprocesses all the images taken by the DES. This reprocessing not only cleans up the “noise” and “junk” and catalogs the stars and galaxies like before, but also “stacks” the images. “Stacking” is the process of overlaying images of the same part of the sky, producing the highest-quality images for scientific analysis. DESDM then releases these clean, stacked, and science-ready images to the DES collaboration so that scientists can make great discoveries with the data!
Data taken with DECam by astronomers during non-DES observing will be processed through the DECam Community Pipeline at the National Optical Astronomy Observatory and made available for analysis.
