Data Warehouse
From Wikipedia: Open notebook is the practice of making the entire primary record of a research project publicly available online as it is recorded. There is no 'insider information'. It is the logical extreme of transparent approaches to research and explicitly includes the making available of failed, less significant, and otherwise unpublished experiments.
The measurements bring in large amounts of data, storage and organizing it is challenging. Each hive sends in data every five minutes - 288 times a day. Each hive inserts over 100,000 rows a year into the Operational Database.
In addition to the measured data, there are other external factors that need to be systematically documented. Metadata includes manipulation data, what mite treatment is used, etc.
The procedures that move the data from the Operational Database to the Research Database:
- Structure the data so that it makes sense to the researcher.
- Structure the data to optimize query performance, even for complex analytic queries, without impacting the operational systems.
- Make research and decision–support queries easier to write.
- Maintain data and conversion history.
- Improve data quality with consistent quality codes and descriptions, flagging and fixing bad data.
The data is:
- cleaned up
- converted (lb <=> kg, Fahrenheit <=> Celsius)
- transformed
- cataloged and
- made available for use by researchers for data mining, online analytical processing, research and decision support
