How unlocking the power of data helps scientists to:
- Unearth new research findings
- Develop better products
- Test new theories
Diamond Light Source is the UK’s national synchrotron. By harnessing the power of electrons, scientists can study anything from fossils to jet engines. Because the amounts of data generated are enormous — close to a petabyte of research data every month — Diamond Light counts on NetApp to make it easily accessible.
Andy Richards, data visionary and head of scientific computing, explains. “Everyday life is affected by everything we do here — from understanding viruses, to drug discovery, all the way through to how to make ice cream taste better.” Here he shares his insights on scientific data management.
Good data management is the key
Richards believes that data management is fundamental to successful research. “Data is everything within Diamond. We operate thirty beamlines, we operate a number of electron microscopes. They all work in different research areas, allowing researchers to look at different problems. But the one thing they all have in common is they all generate increasingly large amounts of data.”
NetApp® technology has enabled Diamond Light to manage and orchestrate the data so that it is stored efficiently and is accessible to the right people with the right clearance. “We use the NetApp all-flash storage such as the EF series for our high-performance data collection from the science experiments. We also use the NetApp FAS series for our more operational data in running the facility,” Richards explains.
Data must do more than consume space and energy
Andy Richards sees data science as one of the most important achievements of the past decade. “It’s transformed the way we turn huge volumes of information into useful predictions and insights,” he says. “Modern technology has allowed scientists to carry out high-resolution genetic sequencing at massive scale. Modern telescopes have transformed astronomy, generating terabytes of raw image data. And technologies like the synchrotron have allowed us to discover new insights at the molecular level.
“Precision data is what allows us to make comparisons, to create models, and to draw meaning from the data that we collect. It is the basis on which hypotheses are tested and either proven or disproven. You can create mountains of data quite easily, but to have value, it must be available for modern scientists to work with complex datasets, algorithms, and metadata. And it has to be easy to use for prediction, exploration, understanding, and intervention.”
Science is a team effort, so data sharing is key
Working at the centre of global research, Richards recognises the critical importance of data sharing. “Even though we are amassing petabytes of data, it has to be shareable so that data scientists and domain experts can collaborate using it.” NetApp has provided Diamond Light with storage solutions that enable them to collect the data in the way they need to and then analyse and help scientists move their data to their home institutions to perform further analysis, whether locally at Diamond or in the cloud.
Data scientists and domain experts need to work closely
According to Richards, data scientists now need to have a much closer working relationship with domain experts. “Data has changed the way we look at things and opened new paths for analysis with new tools. Sure, we can move data from the experiments to our compute cluster for analysis, but it’s not all about automation. It’s also about people having the skills and judgment to interpret results. To do that means having new conversations around new capabilities and trying new techniques or asking new questions that could not have been answered even four or five years ago.”
Researchers from both academia and industry visit the Diamond Light facilities more than 9,000 times annually to use Diamond Light to conduct experiments, assisted by approximately 500 staff members. Because the amounts of data generated from this process are enormous, Diamond Light counts on NetApp technology to make it easily accessible to virtually anyone with a computer. Researchers no longer have to arrive at Oxfordshire with suitcases full of hard drives to capture their data. Now they can access it in the cloud.
Learn more about NetApp All Flash Storage
Diamond Light Source is the UK’s national synchrotron science facility. An important centre for research, it generates close to a petabyte of research data every month, which has led to exciting breakthroughs. It works like a giant microscope, harnessing the power of electrons to produce bright light that scientists can use to study anything from fossils to jet engines to viruses and vaccines. There, scientists use the light to study a vast range of subject matter, from new medicines and treatments for disease to innovative engineering and cutting-edge technology.