🌟Why Flostream?
Flostream introduces the Multi-stream Data Transfer Protocol (MDTP), leveraging UDP-based data algorithms for streamlined efficiency and performance, overcoming limitations found in AWS S3 API.
Understanding How Flostream's Data Transfer Works:
Flostream introduces an advanced method called Flostream Data Transfer Protocol (FDTP), reaching new and improved methods for moving data efficiently and glitch-free. Current industry standards, such as AWS S3 API, involve complex and inefficient methods that often fail, causing significant hassles to users when dealing with large files. Flostream is designed to breakdown data into smaller pieces called flowlets. This helps internet function by minimizing bandwidth requirements for simple, yet large projects, and ensures data gets where it needs to go without lag or risks in being lost during data flow communications. Predominantly tailored for small block file storage, S3’s API leads to non-linear file access patterns, creating operational challenges for standard operating systems that are optimized for sequential data processing.
Flostream's Multi-stream Data Transfer Protocol:
Flostream uses the innovation of Multi-stream Data Transfer Protocol (MDTP) and User Datagram Protocol (UDP) based data algorithms to achieve a more streamlined, efficient alternative in the market. This innovative method is referred to as Flostream Data Transfer Protocol (FDTP), where multi-flow streams and UDP flowlets are broken down into smaller bits, significantly enhancing bandwidth utilization on the network. This method circumvents bit rate limitations set by internet and cloud service providers, enabling enhanced efficiency of data network transmission. Once data arrive at their destination, these segments are systematically reassembled in a reordering buffer. Flostream's custom implementation of Fountain codes closely mirrors the application of Raptor codes. This approach demonstrates Flostream's commitment to adopting and refining advanced coding techniques to optimize data handling within its decentralized storage framework. Fountain codes strategy is implemented in the L1 storage nodes for the data replication process, improving both efficiency and data distribution reliability.
For data integrity and packet loss recovery, Flostream utilizes a combination of convolutional coding and Automatic Repeat reQuest (ARQ) systems. This approach effectively manages the high bit rates of uncompressed data while ensuring its accuracy and consistency during transmission.
In optimizing our network strategy, Flostream chooses to vary the destination port to define each flow, aiming to establish a large path MTU. This allows the transmission of larger packets without fragmentation, resulting in fewer packets per second. This is crucial because some internet and cloud services throttle data based on packets per second. Additionally, fewer packets per second can also reduce the number of system calls in the receiving application, enhancing overall efficiency.
Within each network cell, Flostream endeavors to place instances topologically close to one another, utilizing available tools such as placement groups on AWS. In a data center, and occasionally in external networks, when using Equal-Cost Multi-Path (ECMP) routing, it is expected that packets belonging to a single flow will adhere to the same path. Consequently, a decrease in the sequence number for any flow should be uncommon. If packets arrive out of order, it indicates that the flow has traversed multiple paths.This strategic positioning is essential for optimizing data transfer rates and minimizing latency, ensuring a more efficient and reliable network operation.
Observing burst loss occurring simultaneously across multiple flows might indicate that the network is not solely employing Equal-Cost Multi-Path (ECMP) routing. Alternatively, it could suggest a resource constraint at junctions where several flows converge, which could be either at the sending or the receiving instance.
In recent series of evaluations, the Flostream algorithm was rigorously tested on major cloud platforms, including AWS, Azure, GCP, and Oracle. The primary objective was to ascertain the feasibility of transporting high-bandwidth data, specifically uncompressed video at approximately 2.5 Gbits/s. The results from these tests confirm that using the Flostream Data Transfer Protocol (FDTP), it is indeed possible to achieve this level of data transfer rate consistently across all the tested cloud services. This finding underscores the efficacy of FDTP in managing high-volume data transportation in diverse cloud environments.
Last updated
