Figure 1
EO Data Preservation System.
Figure 2
Master Archive service locations and duties in Europe.
Figure 3
Master Archive infrastructure and high-level data flow between archive centers.
Figure 4
Checks performed during the data ingestion process.
Table 1
Checks performed by the Master Archive during the data ingestion process.
| # | Centre | Operation | Verification |
|---|---|---|---|
| 1 | Main | Ingestion | The list of files is compared to any delivered inventory. |
| 1’ | Main | Ingestion | The structure and the content (repository, datasets, products, files) is compared with the delivery spreadsheet delivered by ESA. |
| 2 | Main | Ingestion | The total number of products is compared to the delivery information. |
| 3 | Main | Ingestion | The actual file MD5 checksum is compared to the value extracted from the product metadata (manifest file or attached checksum file). |
| 3’ | Main | Ingestion | If data is included in a container (zip, tgz, …), the integrity of the container is verified (i.e. container content can be accessed). |
| 4 | Main | Ingestion | After the generation of the zip container, the files are extracted in a scratch directory and checked with respect to the content of the checksum.txt file. This operation is logged for future use by the global verification of the dataset ingestion. |
| 5 | Main | Ingestion | MD5 hash code computed by Quantum from the products are queried from the StorNext database and compared to the DMPC MD5 hash code before products are set in “TAPE” status. |
| 6 | Both | Ingestion | LTO-7 drives apply an automatic verify-after-write technology to immediately check the data as it is being written. |
| 7 | Both | Backend | Both Quantum libraries include EDLM. The Extended Data Life Management feature ensures that tapes are trouble-free (based on tape scan and tape memory analysis). Tapes scan and analysis is performed following predefined policies (max. 4 tapes per day i.e. 7% of a 10 PBytes archive per month using 2 LTO7 EDLM drives). Suspect tapes are automatically copied to new tapes. |
| 8 | Main | Validation | After the ingestion process where data has been verified at product level, a global ingestion verification is performed using the DPMC database information, the initial media inventory and the ingestion process log files. |
| 9 | Backup | Ingestion | The inventory of the tapes sent to the backup centre is retrieved and used for comparison with the copy process performed to copy the products from temporary tapes to ANTF tapes via disk cache. |
| 10 | Backup | Ingestion | The zip container integrity is used to verify that the transferred products have not been corrupted. |
Figure 5
EODAS Service Web Portal.
Figure 6
EODAS overall historical data ingestion status.
Figure 7
EODAS historical data ingestion detail.
Figure 8
EODAS live mission ingestion status.
Figure 9
Front-End Data Circulation.
Figure 10
CBA Volumes, January 2019.
Figure 11
CBA Next Generation Archive.
Figure 12
Data Information System.
Figure 13
DIS Data Provenance.
Figure 14
Archiving Policy.
Figure 15
Data Monitoring.
Figure 16
Data Holdings by Availability.