# DataDrop Intro Decentrally's Sync and Share platform ## Welcome to the forefront of secure digital storage and privacy. We're about to embark on a journey through the advanced processes and technologies that safeguard your data with DataDrop. Our platform integrates seamlessly with Filecoin's decentralized network, providing a robust, secure, and efficient method of managing your digital assets. From the moment you upload your file to the instant you may choose to erase it from existence, every step is designed with the utmost security in mind. Let’s dive into how DataDrop ensures that your data is not only protected but also remains entirely under your control. # DataDrop Architecture ### Overview Our Sync & Share platform a.k.a. DataDrop, empowered by Filecoin’s decentralized storage, transforms data management into an art. Imagine effortlessly managing all kinds of data, seamlessly moving it across clouds with unparalleled freedom. And with Filecoin’s proof of storage, we’re not just talking about storage; we’re talking about secured, verified storage that changes the game. This is data storage, reimagined and supercharged. Welcome to the future, where your data flows freely and securely, exactly as you need it to.
### Component description * **The User, ISV, Integrator:** At the heart of our ecosystem, users, Independent Software Vendors (ISVs), and Integrators leverage our platform to streamline their operations, enabling seamless data management and collaboration across various environments. * **S3 object interface:** Utilizing the familiar S3 object storage interface, our platform offers compatibility and ease of integration, ensuring users can manage their data with the tools they know and trust. See : [S3 Cloud Storage](/s3-cloud-storage-intro) * **Sync & Share platform:** Our core feature allows for effortless synchronization and sharing of data across devices and cloud environments, enhancing productivity and collaboration. * **Decentrally:** Powered by Decentrally, our platform embodies flexibility and security, providing a decentralized approach to data storage and management that breaks free from traditional limitations. * **AWS S3, Google cloud, Azure blob:** We support seamless integration with major cloud providers including AWS, Google Cloud Platform, and Azure, offering unparalleled flexibility in data storage and movement. * **Filecoin:** Leveraging Filecoin’s decentralized storage network, our platform ensures data is stored securely and reliably, with proof of storage capabilities offering an added layer of verification and trust. * **Storage Providers:** Our ecosystem includes a wide range of storage providers, allowing users to choose the best storage solution for their needs, from traditional cloud storage to decentralized options. * **Proofs:** With Filecoin’s innovative proof mechanisms, such as proof of storage, our platform guarantees the integrity and availability of data, ensuring it is stored exactly as intended across a distributed network. # How does DataDrop work ### Graphical Data flow

DataDrop - Data flow

### Data flow 1. **Upload:** The process begins with the upload of a file to the system. 2. **In-transit Encryption:** As the file is being uploaded, it is encrypted to protect its contents while it is being transmitted over the network. 3. **Encryption:** Once the file reaches its destination, it undergoes another layer of encryption for additional security while at rest. Also read [Data security](/datadrop-intro/datadrop-data-security) 4. **Data Chunking:** The encrypted file is then divided into smaller chunks, which makes it easier to manage, store, and retrieve. Also read [Data security](/datadrop-intro/datadrop-data-security) 5. **Host & Store:** These chunks are then hosted and stored across the decentralized network of storage nodes provided by Filecoin. 6. **Data Lookup:** When the file is requested for download, the system performs a lookup to find all the chunks that constitute the original file. 7. **Reassembly:** The system reassembles the chunks back into the original file format. 8. **Decryption:** Before the file is provided to the downloader, it is decrypted to make it accessible. 9. **In-transit Encryption:** As the file is being transmitted back to the requester, it is encrypted again to ensure security during transit. 10. **Download:** Finally, the file is downloaded by the requester, at which point it can be decrypted and accessed. This process as part of the DataDrop platform highlights how it integrates with Filecoin’s decentralized network to provide enhanced security, cost efficiency, and reliable data availability for cloud storage. # DataDrop Data security ### At-rest encryption Within the DataDrop platform, when a file completes its journey to the designated storage point, it engages in a critical security practice known as **at-rest encryption**. This is a pivotal step where the file is encrypted using sophisticated cryptographic algorithms to ensure that even if unauthorized access to the storage is obtained, the contents of the file remain unintelligible and protected. The at-rest encryption provides an impervious shield, securing the data against threats such as data breaches and unauthorized disclosures. ### Data chunking As part of its fortified security strategy, the platform implements **data chunking**. Post encryption, the file is disassembled into smaller, more manageable pieces. These fragments, akin to a complex jigsaw puzzle, are then distributed across the Filecoin decentralized network. This dispersion means that the pieces of your file live in separate, distinct locations, rendering the task of unauthorized reassembly virtually impossible. Should an intruder access one piece, without the rest and the unique keys, it remains an unsolvable riddle. ### Encryption techniques The **encryption techniques** employed are industry-standard protocols such as AES (Advanced Encryption Standard) with a 256-bit key for robust security and RSA for secure key exchange, ensuring that data is safeguarded with the same level of encryption that is trusted by banks and government agencies. RSA provides a secure method of exchanging the keys necessary for decrypting the data chunks. Since the key to decrypt the data is separate from the data itself, and because knowledge of the system's distribution pattern is required, reassembling the file without authorized access is akin to finding a needle in a haystack – not just once, but for every single piece of the file. ### Final thoughts This intricate dance of chunking and encryption, paired with Filecoin’s decentralized distribution, ensures that even if one were to obtain a chunk, without the decryption key and the knowledge of the chunk's relation to others, the data remains an enigma. This system exemplifies how DataDrop doesn’t just lock the vault – it scatters it across the globe, with each piece locked separately. # Deletion handling \ In the realm of digital security and privacy, the irreversible deletion of data is just as critical as its protection. DataDrop upholds this principle through meticulous processes that ensure once a user decides to erase a file, it becomes irretrievable. The platform's deletion protocol is comprehensive and multi-layered, addressing every element that could possibly be used to reconstruct the data. ### Metadata When a deletion command is initiated, the system first purges the **metadata** associated with the file. This metadata includes all information about the file's structure, its location across the decentralized network, and any other descriptive details that could be used to identify or locate the chunks of data. ### Encryption keys Simultaneously, DataDrop annihilates the **encryption keys**. These keys are the only means to decrypt the data, translating it from an unintelligible state to a readable one. Once the keys are destroyed, the data, even if somehow located, remains an indecipherable cipher—permanently locked within its encrypted shell. ### Chunking information Furthermore, the platform eradicates the **chunking information**—the blueprint illustrating how the file is fragmented and scattered across the storage network. Without this blueprint, reconstructing the data from its dispersed chunks becomes a feat with astronomical improbability. ### Final Thoughts The absence of these critical components—metadata, encryption keys, and chunking information—renders the data not just inaccessible but essentially non-existent. It's akin to dispersing ashes into the sea; once done, the original form is lost forever. By obliterating these elements, DataDrop guarantees that deleted data, along with its potential to be accessed or recovered, is expunged from the digital world, assuring users that their digital footprint can be effectively and securely erased. # Governance - GDPR - SOC Data tracking visibility is a crucial aspect of data governance and security. It allows users to monitor the lifecycle of their files, including creation, modification, access, and transfer events. In the example shown, there is a dedicated area for “File Activity” that lists recent actions taken on documents, such as moving, modifying metadata, or creating a new document, along with timestamps and the identity of the individual who performed the action. This transparency serves several important purposes: **Audit Trail**: It offers a comprehensive audit trail, essential for regulatory compliance and for maintaining records of data handling. **Security Oversight**: By keeping track of who accessed or modified a file and when, it helps in detecting unauthorized access or changes, thereby bolstering security. **Collaboration Efficiency**: In collaborative environments, it enables team members to see the most recent actions taken on a document, which can aid in coordination and prevent redundant efforts. **Version Control**: It assists in version control by providing a history of modifications, ensuring that users are always working on the latest version of a document. **Accountability**: By attributing actions to specific individuals, it ensures accountability within teams and organizations. ## GDPR For **GDPR**, which emphasizes data privacy and user rights, having a detailed audit trail of data interactions is essential. It enables organizations to: **Monitor Data Processing**: Track and document processing activities, which is a requirement under GDPR. **Data Access and Rectification**: Quickly identify who accessed or modified personal data, helping address the right to access and the right to rectification. **Security Measures**: Demonstrate the implementation of appropriate security measures by logging access and changes to data. **Breach Notification**: Assist in breach detection and subsequent notification procedures by providing timely records. ## SOC (2) For **SOC**, particularly SOC 2 which deals with the management of customer data based on five “trust service principles” — security, availability, processing integrity, confidentiality, and privacy, data tracking visibility is crucial as it: **Security**: Enhances security measures by providing logs that can be analyzed for potential unauthorized access or anomalies. **Availability**: Ensures data is accessible as needed, with logs reflecting the history of data access. **Processing Integrity**: Shows a clear record of all actions taken on data, ensuring processing is valid, timely, complete, and authorized. **Confidentiality and Privacy**: Records all instances of data access, modification, and deletion, which is vital for maintaining confidentiality and privacy. By maintaining detailed records of data interactions, organizations can respond more effectively to audits, provide evidence for compliance, and improve their overall data governance strategy. Overall, data tracking visibility empowers users to maintain control over their data, ensure the integrity of their information, and enhances the collaborative process by providing a clear and accountable record of file interactions. # S3 Cloud Storage Intro Decentrally's S3 Cloud Storage Buckets provide a solution for storing and delivering large amounts of data within an S3-compatible object storage system. Each Storage Bucket acts as a private vault for storing and distributing files. The decentralized design ensures enhanced privacy, ensuring that only the user has access to their data. # S3 Cloud Storage Architecture ### Overview Our S3 Cloud Storage infrastructure, empowered by Filecoin’s decentralized storage, transforms data management through an easy to use S3 compatible API.\ \ Imagine effortlessly managing all kinds of data, seamlessly moving it across clouds with unparalleled freedom. (Some features are optional, see [DataDrop](/)) And with Filecoin’s proof of storage, we’re not just talking about storage; we’re talking about secured, verified storage that changes the game. This is data storage, reimagined and supercharged. Welcome to the future, where your data flows freely and securely, exactly as you need it to.
### Component description * **The User, ISV, Integrator:** At the heart of our ecosystem, users, Independent Software Vendors (ISVs), and Integrators leverage our platform to streamline their operations, enabling seamless data management and collaboration across various environments. * **S3 object interface:** Utilizing the familiar S3 object storage interface, our platform offers compatibility and ease of integration, ensuring users can manage their data with the tools they know and trust. ([Tools](/s3-cloud-storage-intro/sdks-and-clis)) * **Decentrally:** Powered by Decentrally, our platform embodies flexibility and security, providing a decentralized approach to data storage and management that breaks free from traditional limitations. * **AWS S3, Google cloud, Azure blob:** We support seamless integration with major cloud providers including AWS, Google Cloud Platform, and Azure, offering unparalleled flexibility in data storage and movement. * **Filecoin:** Leveraging Filecoin’s decentralized storage network, our platform ensures data is stored securely and reliably, with proof of storage capabilities offering an added layer of verification and trust. * **Storage Providers:** Our ecosystem includes a wide range of storage providers, allowing users to choose the best storage solution for their needs, from traditional cloud storage to decentralized options. * **Proofs:** With Filecoin’s innovative proof mechanisms, such as proof of storage, our platform guarantees the integrity and availability of data, ensuring it is stored exactly as intended across a distributed network. # How does S3 Cloud Storage work ### Graphical Data flow

S3 Cloud Storage - Data flow

### Data flow 1. **Upload:** The process begins with the upload of a file to the system. 2. **In-transit Encryption:** As the file is being uploaded, it is encrypted to protect its contents while it is being transmitted over the network. 3. **Encryption:** Once the file reaches its destination, it undergoes another layer of encryption for additional security while at rest. Also read [Data security](/s3-cloud-storage-intro/s3-cloud-storage-security) 4. **Data Chunking:** The encrypted file is then divided into smaller chunks, which makes it easier to manage, store, and retrieve. Also read [Data security](/s3-cloud-storage-intro/s3-cloud-storage-security) 5. **Host & Store:** These chunks are then hosted and stored across the decentralized network of storage nodes provided by Filecoin. 6. **Data Lookup:** When the file is requested for download, the system performs a lookup to find all the chunks that constitute the original file. 7. **Reassembly:** The system reassembles the chunks back into the original file format. 8. **Decryption:** Before the file is provided to the downloader, it is decrypted to make it accessible. 9. **In-transit Encryption:** As the file is being transmitted back to the requester, it is encrypted again to ensure security during transit. 10. **Download:** Finally, the file is downloaded by the requester, at which point it can be decrypted and accessed. This process as part of the S3 Cloud Storage infrastructure highlights how it integrates with Filecoin’s decentralized network to provide enhanced security, cost efficiency, and reliable data availability for cloud storage. # S3 Cloud Storage security ### At-rest encryption Within the S3 Cloud Storage platform, when a file completes its journey to the designated storage point, it engages in a critical security practice known as **at-rest encryption**. This is a pivotal step where the file is encrypted using sophisticated cryptographic algorithms to ensure that even if unauthorized access to the storage is obtained, the contents of the file remain unintelligible and protected. The at-rest encryption provides an impervious shield, securing the data against threats such as data breaches and unauthorized disclosures. ### Data chunking As part of its fortified security strategy, the platform implements **data chunking**. Post encryption, the file is disassembled into smaller, more manageable pieces. These fragments, akin to a complex jigsaw puzzle, are then distributed across the Filecoin decentralized network. This dispersion means that the pieces of your file live in separate, distinct locations, rendering the task of unauthorized reassembly virtually impossible. Should an intruder access one piece, without the rest and the unique keys, it remains an unsolvable riddle. ### Encryption techniques The **encryption techniques** employed are industry-standard protocols such as AES (Advanced Encryption Standard) with a 256-bit key for robust security and RSA for secure key exchange, ensuring that data is safeguarded with the same level of encryption that is trusted by banks and government agencies. RSA provides a secure method of exchanging the keys necessary for decrypting the data chunks. Since the key to decrypt the data is separate from the data itself, and because knowledge of the system's distribution pattern is required, reassembling the file without authorized access is akin to finding a needle in a haystack – not just once, but for every single piece of the file. ### Final thoughts This intricate dance of chunking and encryption, paired with Filecoin’s decentralized distribution, ensures that even if one were to obtain a chunk, without the decryption key and the knowledge of the chunk's relation to others, the data remains an enigma. This system exemplifies how our S3 Cloud Storage solution doesn’t just lock the vault – it scatters it across the globe, with each piece locked separately. # SDKs & CLIs # CLI ## Prerequisites Before you can use our S3 Cloud Storage with the AWS CLI, you must install the AWS CLI and configure it. 1. [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - ref docs.aws.amazon.com 2. [Configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) - ref docs.aws.amazon.com After you have installed and configured the AWS CLI locally, the following sections will show you how to use the CLI. For complete examples, [see the AWS CLI S3 examples](https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html). ## List buckets ``` aws s3 --endpoint-url https://s3proxy.decentrally.cloud ``` ## List objects ``` aws s3 --endpoint-url https://s3proxy.decentrally.cloud ls s3://bucket ``` ## Upload object ``` aws s3 --endpoint-url https://s3proxy.decentrally.cloud cp /path/to/file s3://bucket ``` ## Get object ``` aws s3 --endpoint-url https://s3proxy.decentrally.cloud cp s3://bucket/file /path/to/file ``` ## Delete object ``` aws s3 --endpoint-url https://s3proxy.decentrally.cloud rm s3://bucket/file ``` ## Delete multiple objects ``` aws s3 --endpoint-url https://s3proxy.decentrally.cloud rm s3://bucket/ --recursive ``` ## Copy object ``` aws s3 --endpoint-url https://s3proxy.decentrally.cloud cp s3://bucket/file s3://bucket/file ``` # Golang ### 1. Install the Dependencies To install the project dependencies, simply run the following command in your terminal: ```go go mod init aws_s3 go get github.com/aws/aws-sdk-go-v2/service/s3 ``` The following commands show how to retrieve the standard set of SDK modules to use in your application. ```go go get github.com/aws/aws-sdk-go-v2 go get github.com/aws/aws-sdk-go-v2/config ``` ### 2. Setup the Environment We will be setting up two environment variables, `ACCESS_KEY` and `SECRET_KEY`, which are required to access the S3 Cloud Storage service. Assign the values of these variables to the values of your credential that you received. ```go export ACCESS_KEY= export SECRET_KEY= ``` ### 3. Minimal code example ```go package main import ( "context" "log" "os" "strings" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/s3" ) func main() { resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) { return aws.Endpoint{ URL: "S3_ENDPOINT_URL", SigningRegion: "us-east-1", }, nil }) credentials := aws.CredentialsProviderFunc(func(ctx context.Context) (aws.Credentials, error) { return aws.Credentials{ AccessKeyID: "YOUR_ACCESS_KEY_ID", SecretAccessKey: "YOUR_SECRET_ACCESS_KEY", }, nil }) // Load config cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithCredentialsProvider(credentials), config.WithEndpointResolverWithOptions(resolver), ) if err != nil { log.Fatal(err) } // Create an Amazon S3 service client s3Client := s3.NewFromConfig(cfg, func(o *s3.Options) { o.UsePathStyle = true }, ) filePath := "YOUR_FILE_PATH" // Get file name from filePath path := strings.Split(filePath, "/") fileName := path[len(path)-1] file, openErr := os.Open(filePath) if openErr != nil { log.Fatal(openErr) } defer file.Close() _, putErr := s3Client.PutObject(context.TODO(), &s3.PutObjectInput{ Bucket: aws.String("YOUR_BUCKET_NAME"), Key: aws.String(fileName), Body: file, }) if putErr != nil { panic(putErr) } log.Println("Successfully uploaded object") } ```