Huawei’s contribution

The year 2012 also brought a new contributor into the Platform Competency Centre as Huawei joined covering the rapidly expanding area of cloud storage. Since CERN is faced with rapidly expanding requirements in storage scalability and performance for physics and infrastructure data, these new technologies and implementations are expected to complement the existing service offerings at CERN and may play a role in lowering the service cost by consolidating services through novel deployment approaches for very large-scale disk storage. Huawei on the other side is expanding their product offering with new highly scalable cloud storage products aimed at providing redundant, large-scale cloud storage closely following the S3 interfaces at a reduced cost of ownership.

In order to evaluate the benefits of cloud storage in the environment of science data storage and analysis, a comprehensive test plan has been agreed and executed throughout the year focusing on scalability and performance. Hardware basis for the evaluation is a storage system consisting of 384 hard drives with associated low power processors interconnected in a highly redundant cluster which is connected via two 10 gigabit network connections to client nodes in the CERN computer centre.

After the rapid commissioning and system testing at CERN by Huawei storage engineers in January, the joint tests quickly confirmed that the system was compliant to a S3 protocol test suite, which had been developed at CERN. The team then proceeded to evaluate the performance and scalability of individual product components such as the distributed meta-data storage, which are of crucial importance for data analysis use cases characterised by rapid access to a large number of different files. The project team confirmed the expected scalability up to 18,000 file downloads and 1,400 file uploads per second. A few smaller issues have been quickly identified and resolved via weekly conference calls with the main product architect and the help of on-site Huawei engineers.

In collaboration with the ROOT team at CERN, direct access to S3 based storage from production experiment applications could be demonstrated with satisfying first performance results, taking into account that cloud storage is not usually optimised for random access use cases. In a final testing phase the number of storage clients has been ramped up until the available network bandwidth of 20 gigabit per second was saturated serving 336 concurrent clients and the project team could confirm close to linear scalability with increasing number of clients and storage resources. CERN and Huawei are now considering scenarios for how to leverage the results achieved as part of a future joint programme.