Overview
The Storage Developer Conference (SDC) 2020 was held virtually on 22rd and 23th of September. It was organized by the Storage networking Industry Association (SNIA), sponsored by SODA Foundation, and attracted 192 speakers from various companies across the industry.
The two-day conference had 7 keynote presentations by SINA, Dell, IBM, Amazon, and other organizations; over 130 breakout sessions covering 18 topics; and 9 live sessions for enterprise discussions through Microsoft Teams and Zoom. In addition, participants interested in specific topics engaged on Slack channels for Q&A.
Breakout session Topics included:
- Blockchain
- Container storage
- Computational Storage
- Cloud Storage
- Data protection and data security
- File Systems
- Key Value
- NVMe
- NVMe-oF
- Orchestration
- Persistent memory
- SMB
- Storage Architecture
- Storage Networking
- Storage Resource Management
- Solid State Storage Solutions
- Storage Performance/Workloads
- Zoned Storage
Talks on Orchestration
Topic: SODA – One Data Framework, Infinite Possibilities
Rakesh Jain, Senior Technical Staff Member, IBM Research, IBM
Anjaneya Chagam, Cloud Architect, Intel Corporation
In this talk, we introduced SODA Foundation, its history, use cases to addresses, roadmap and mechanism for other open source projects to become part of the community.
SODA Foundation is a transformation of the Linux Foundation OpenSDS technical project. With the support of members, SODA Foundation was conceived and officially launched on June 29, 2020. It is designed to address types of workloads such as artificial intelligence, machine learning, activities and operations for storage types, and aims to create a unified data framework, form meaningful standardization, and work with the ecosystem to provide qualified, high-quality solutions.
Through the introduction of SODA Foundation’s Framework, architecture, governance, programs currently in implementation, and use cases, the organization aims to jointly develop projects, standardization, and certifications through close collaboration with vendors, system integrators, cloud service providers, standards organizations, and consortiums.
The three missions of SODA Foundation are:
- Foster an ecosystem of open source data management and storage software for data autonomy
- Offer a neutral forum for cross-projects collaboration and integration
- Provide end users quality end-to-end solutions
Rakesh and Anjaneya concluded the talk with a demonstration of multi cloud data management for participants to gain a better understanding of SODA Foundation.
Topic: Unified heterogeneous storage monitoring : Is delfin a way forward?
Najmudheen CT, Architect Huawei, Maintainer SODA Foundation, Huawei Technologies India Pvt Ltd
Masanori Itoh, Principal Researcher, Toyota Motors, TSC Member SODA foundation, Toyota
In this talk, we discussed how unified data models can be built for different concepts and KPIs across multi vendors. We also discussed the architecture tenets of delfin and provided the technical insights.
Most companies are facing the challenges of the heterogeneous infrastructure: the need to monitor different devices and attempt to create a coherent end-to-end solution from multiple vendors. The solutions offered by are not unified, cooperative or open.
What if storage administrators had a unified open solution to optimize service levels: unified monitoring and alert, capacity trending and target analysis, quick spotting of performance bottlenecks?
Najmudheen and Masanori explored with participants the SODA Foundation delfin project, architecture and roadmap. delfin is a SODA Infrastructure Manager project which provides unified, intelligent and scalable resource management, alert and performance monitoring, designed to achieve the goal of making a heterogeneous layer for managing data from edge to core.
Topic: Predictive analysis of storage health and performance for heterogeneous environment
Najmudheen CT, Architect Huawei, Maintainer SODA Foundation, Huawei Technologies India Pvt Ltd
Eric Xu (Xu-lin), Architect Huawei, Maintainer, SODA Foundation
This talk presents a case study of how “telemetry” and “anomaly detection” can be put together to solve data centre predictive analysis problems using available open source solutions, especially for a heterogeneous environment with different vendor storages or application clients
Data center administrators want to improve the effectiveness of their data center management. However, they face complex infrastructure (switches, servers, storage etc.), different technologies to manage, low resource utilization, inefficient fault detection, and no recycling of resources.
Most companies in the DCIM Maturity Model are at Level 3, the proactive phase. The goal is Level 5, the self optimizing and autonomic phase, where multiple IT and infrastructure subsystems are integrated and optimization happens in near real time. Telemetry is therefore required because it simplifies storage resource management, provides visual analysis monitoring and software reporting to each vendor, and enables monitoring of different capability metrics, performance and configuration.
Najmudheen and Eric presented the SODA Foundation delfin project for storage infrastructure management to help participants understand how SODA Foundation focuses the data collection and configuration of infrastructure for any OEM platform to help optimize data center management.
The concept of anomaly detection was elaborated in one of the SODA Foundation’s core projects, which could help with implementation of intelligent monitoring through experts with Kafka (or similar) and standard Machine Learning algorithms. The advantage is that any machine learning based on leverage can be integrated into the framework for data analysis and prediction.
At the end of the presentation, Eric demonstrated how SODA Foundation brings telemetry and anomaly detection together to solve data centre predictive analysis problems.
Topic: Resilient Workflow Automation in a Hybrid Cloud environment
Ashit Kumar, Architect, SODA Foundation, Huawei
Joseph Vazhappilly, Senior System Architect, SODA Foundation, Huawei
Ashit and Joseph presented an open source orchestration engine agnostic framework. This project, developed under SODA Foundation, can support heterogeneous orchestration engines in a pluggable extension way, provides custom workflow development and deployment agnostic to those orchestration backends.
Every service in the on-prem, cloud or hybrid environments is a sequence of specific operations to realize a specific use case. Hence building custom workflows for each use case is demanding. If it is with a framework that is orchestration engine agnostic, then it is really compelling!
Based on the definitions of workflow and orchestration, the Resilient Workflow Automation (RWA) working principle was explained: it can be used anywhere; custom workflows can be defined; the workflow orchestration engine is agnostic; plug and play with any engine; adaptability of the framework.
Ashit demonstrated two interfaces of the SODA project orchestrations provided: SODA Foundation’s dashboard (provisioning and migration of two workflows), and the CLA. How the service catalog manager and the workflow manager work was also demonstrated.
Finally, Joseph explained how to use RWA through the prototype demo and its benefits: the open source allocation framework can be customized according to the choreography engine to define actions; it can be developed and protected by the community; and no vendor lock-in
Topic: Autonomous Data Management at Edge: Challenges and possibilities.
Sanil Kumar D, Chief Architect, TOC, Head SODA India, SODA Foundation, Huawei Technologies India Pvt Ltd
Vinod Eswaraprasad, Chief Architect, Global Head of Cloud & Platform Practice, Wipro
In this session, Sanil and Vinod discussed the data management requirements and challenges at the Edge, and opened the forum to build an open and autonomous data management framework for Edge.
Sanil and Vinod introduced the requirements, challenges, and needs in Edge development. To illustrate the architecture and proposal, they used heterogeneous storage management solution models from SODA Foundation projects.
Edge advantages:
- It reduces costs on the server by avoiding transfer of the data from on-prem to cloud or it doesn’t need to go through multiple hops, from where the data is generated.
- There is no need to send from the Edge to different locations, data movement is reduced, and more places can be kept tested in one process, so security is improved.
- It is real-time, thus the impact of ultra-low latency is minimized.
- The Edge network can run independently without having to connect to the cloud.
Key requirements of Edge Computing:
- Maximize the computing at Edge
- Offline scenarios and communications
- Security & Privacy
- Scalability-platform and clusters
- Device life cycle & management
- Data & data analytics
- Remote management & visualization
- Efficient energy management
The importance of Edge storage is that most enterprises adopt a multi-cloud strategy, and the data must be transferred from different cloud providers who have different devices, so the whole structure is complex. In addition, centralized to decentralized storage is shifting to distributed storage. As a result, the complexity of data management and infrastructure increases the edge of data, but also presents different requirements and challenges.
Challenges:
- Multi-source data generation: Data operations come from multiple different sources from the edge to the core, resulting in complex data collection, coordination and the whole pipeline, increasing the difficulties in processing and analysis.
- Heterogeneous Data Storage: Different types of storage devices add an additional complexity during data storage.
- Data distribution: This requires a seamless peer-to-peer network and the available data should be at the edge of any node and the computing infrastructure. The distributed nature of the edge storage network and the ability to ensure data ownership and consistency across nodes require a new look at deploying data services on edge.
- To avoid the communication latency that happens with a poor cloud: The heterogeneity of storage nodes at the edge requires us to provide caching strategies for different data stores to ensure consistency of the cached data across nodes.
- Allowing processes to be stored offline: Data to be retrieved can be caused by a variety of reasons, so the ability to aggregate data from devices, sensors, and offline environments is highly demanded.
- Data mobility: How to ensure that data can span peer nodes, ensuring that any application and any consumer can move their orientation from an edge location.
What we need:
- Heterogeneous data framework
- Open
- Vendor agnostic
- Platform agnostic
- Distributed
- Low resource
- Extensible or shrinkable
- Standardized
A possible solution: A distributed, heterogeneous data management platform to meet the challenge of EDGE. An open platform can manage this heterogeneous data, especially when it comes to distributed data management. Allowing multiple IPs, cross-community efforts, and cross-vendor storage portability, a scalable solution prevents more devices from being added and removed, and then standardized.
SODA Foundation can provide a solution by One Data Framework. The KubeEdge project was demonstrated to show that SODA starts from Edge Computing platform, where distributed and heterogeneous data management platform can be the solution of Edge.
Talk on Container Storage
Topic: One CSI plugin for All? Experimenting heterogeneous storage with single CSI plugin for Kubernetes
Sushantha Kumar, System Engineer, Huawei Technologies India Private Limited
Mohammad Asif Siddiqui, System Engineer, Huawei Technologies
In this session, the authors showcased how to ease the overall management and configurations, by having one Container Storage interface (CSI) plugin to handle multiple existing CSI Drivers.
Kuberetes is a popular and well grown container orchestration system available today. With the growing number of application deployment, the persistent volume aspect for the application has become one key area. Container Storage interface (CSI) is the popular and conventional way by which different backend systems provision the storage for kubernetes applications. There are numerous CSI drivers available to get the particular storage systems glued to the containerized applications. In an ecosystem with the presence of heterogeneous storage backends, managing the multiple CSI drivers for different storages is a challenge and also an opportunity.
Sushantha explained how Kubernetes and CSI interacted. The demo showed a current operational scenario of the SODA CSI plug-in. This is the SODA CSI plug-in that provides unified control over the traditional cloud and cloud local environment, supports configuration file management, and enables the cluster to map the storage specification template of the Kubernetes storage class. It allows users to define their own CSI plug-ins, which in turn, requests the SODA server components.
Another demo explained how the two SODA CSI plug-ins play design options work and the goals they were trying to achieve. Mohammad showcased the step forward to add multiple data management features (like snapshot, replication and more) from SODA Foundation (under Linux Foundation) open source projects along with CSI with this integration model.
In the conference, SODA Foundation speakers contributed six presentations in data storage management, particularly orchestration. Demonstrations on core projects outlined how the solutions are addressing current challenges. Participants also gained an understanding of the development of SODA Foundation, the management model, organizational structure, the three missions, as well as the projects under implementation and future projects planned.
One Data Framework, Infinite Possibilities
Interested in participating in SODA project development?
Take an active role in supporting the growth and evolution of the SODA ecosystem.
About the Sponsor and the Organizer of the SDC 2020 Conference
SODA Foundation is an open source project within the Linux Foundation that aims to foster an ecosystem of open source data management and storage software for data autonomy. It aims to include multiple open source projects from different communities which are related to data management. The SODA data framework is part of this effort to enable data mobility, data protection, data security, data lifecycle, and more for cloud native, virtualization, and other environments. In this talk, we will introduce SODA Foundation, its history, use cases to addresses, roadmap and mechanism for other open source projects to become part of the community.
The Storage Networking Industry Association (SNIA) is a non-profit organization made up of member companies spanning information technology. A globally recognized and trusted authority, SNIA’s mission is to lead the storage industry worldwide in developing and promoting vendor-neutral architectures, standards, and educational services that facilitate the efficient management, movement, and security of information.
###
