In the past three weeks, members of the Filecoin community participated in the first phase of the Slingshot competition, storing 450+ TiB of data through more than 350,000 storage transactions on the network. To help the projects in the competition, many Filecoin community members gave presentations on different topics related to storing data and building on Filecoin. However, the value of these conversations is not limited to slingshots. They are perfect for anyone who wants to work with Filecoin!
In this article, we will reintroduce some of the lectures and panel discussions to help you quickly start storing and building on Filecoin. You can view the complete list of Filecoin events, including past records and future events.
Master Class: Introduction to Powergate
Andrew Hill, co-founder and CEO of Textile, chaired the meeting. Textile provides tools that make it faster and easier for developers to build decentralized applications on IPFS and Filecoin. In this master class, Andrew discussed how to start and use Textile Powergate, which is a library for managing storage and retrieval on Filecoin and IPFS networks.
You can run Powergate on your own computer or use one of Textile's hosted instances, which allows you to store a single data in the IPFS and Filecoin network with just a few commands?? Any one! It also classifies data retrieval requests between the two networks, where IPFS is generally more suitable for more frequent "hot" storage, while Filecoin is generally more suitable for less frequent "cold" storage.
More generally, Powergate is one of the easiest ways to store on Filecoin, handling many details about making Filecoin transactions. It has Javascript and Golang clients, as well as community-developed Python clients. First, check the documentation here.
Textile also produces Buckets, which are S3-like cloud storage solutions with Filecoin capabilities. You can watch "Introduction to the Barrel Master Course of Textiles" here.
Benefits of using Powergate:
Ensure that the data stored on Filecoin is easily accessible on the IPFS network for quick retrieval.
Handle long-term storage transaction management, including automatic renewal and repair.
Use network indexing to improve miner selection and transaction creation.
Manage the Filecoin wallet address of one or more users.
Easily configure, connect and deploy Powergate, Lotus and IPFS together. V丨hugoschan
Andrew's main skills using Powergate:
1. Start with Localnet. You can start testing your ideas with Filecoin in 5 minutes. No FIL is required, end-to-end transactions can be performed within one minute, and there are no network problems.
2. Follow the leader. Many great teams have been established on Powergate. So far, we have seen that they are very helpful. For the early team, this may be very valuable.
3. Start small. After you leave Localnet and start trading on the real-time network, please be careful not to trigger the entire pipeline in the first place. Choose some GiB, start a transaction, check the life cycle, and then proceed with a larger transaction.
Master class: prepare large data sets for file currency storage
Protocol Labs IPLD team members Rod Vagg, Mikeal Rogers, and Chris Hafey talked to Slingshot participants on how to store large data sets in Filecoin. IPLD is a key component of Filecoin, which helps data structures across content addressing networks to achieve interoperability.
Rod first gave the basic introductory knowledge of IPLD, that is, "content addressing data structure with IPLD". He discussed Merkle trees, DAG, CIDS, codecs, variability, and other key concepts that support Filecoin and many other content-addressable data structures.
Chris and Mikael's presentation "Using Dumbo Drop to prepare large data sets for Filecoin storage" introduced the general method, architecture, and lessons of the Filecoin Dumbo Drop project. The goal of Dumbo Drop is to process a large amount of open data for Filecoin in a short time. So far, the project has processed more than 3PB of data.
Some lessons they learned:
1. AWS is not 100% reliable on this scale-random failures will happen!
2. In this case, there are some quirks about using S3. S3 limits prefix-based performance, and S3 list objects are slow and unreliable for a large number of objects.
3. Compared with EC2, Lambda can be more flexible and easier to use. Lambda keeps getting cheaper and easier to use. However, customizing Lambda can be tricky.
4. Computing is cheap, but storage is expensive.
Important note for Filecoin users:
1.Rod: "Understand the data primitives, learn the basics of IPLD and the great potential hidden in the Merkle DAG data structure."
2. Chris: "Consider building your own pipeline from the same library used by IPFS/Lotus."
3. Mikael: "You can store more than just files:)"
How we built it: The Package Manager registry on Filecoin
Brian Hoffman, the co-founder and CEO of OB1, discussed their team's 5MB project, which aims to store and retrieve the package manager registry from Filecoin. OB1 was established by a small team in 2015. Before that, they worked on OpenBazaar in their spare time for one year. They saw that the concept of decentralized transactions gained enthusiastic support in the Bitcoin community, and decided to accept venture capital to build a company and hire a development team to make OpenBazaar something that can revolutionize e-commerce.
Brian discussed the motivation and process of building 5MB. He introduced the technical nuances that the project must address when storing versioned data and small files on Filecoin and building a UI for browsing the data.
Brain also introduced us to the 5MB architecture, which utilizes Textile Powergate: V丨hugoschan
Phase 1: Extract, where 5MB is extracted from the package repository to Amazon EBS, and the data is divided into different data sets for further processing
Phase 2: Processing, where the Golang processor server divides the data into IPFS, evaluates the size and structure of the data, and decomposes directory objects into buckets.
Phase 3: Archive and retrieval, where the data storage area is pushed into Filecoin through Powergate and can be accessed through the UI (a modified version of the IPFS GUI)
Three reasons why Brian established Filecoin:
Sustainable data ecosystem: We need a way to incentivize data storage infrastructure in a sustainable way with reliable large-scale data sets.
Required tools and applications: Developers need better and more diverse tools to take full advantage of the network, and users want easy-to-use applications to help them access the data they need.
IPFS has established a reputation: We have used IPFS for many years and understand its advantages and disadvantages. The technology works and we are excited about Filecoin.
How we built: slate
Slate is a storage application and Filecoin web client that allows you to store images, audio, video, and data through a graphical interface or API. Jim Lee, one of the co-founders of Slate, discussed what Slate is, its evolution from a simple Filecoin client to a powerful media sharing product and its future development. He hopes to share lessons and insights from becoming an early adopter of applications on top of Filecoin.
The design of the product considers privacy to a certain extent; no one can see which CIDs are uploaded from Slate to the Filecoin network, and the storage area where the files are stored is encrypted before uploading the data. The team plans to establish an offline mode where your files are all private files.
Jim introduced the current state of the application while introducing the background of the development and innovation process. He talked about the feeling of using Filecoin in the early days, how the use of Textile technology made things much easier, the team's ideas and product development, and his experience in building applications on cutting-edge protocols.
Jim also showed the basics of Slate (for example, how basic file uploads work in the browser), and showed examples of the potential of applications such as Slate, and demonstrated how to use Slate to build simple applications to store data. He also created a very simple application using Powergate and Slate components to send and receive Filecoin.
Topic: Widely adopted road
Projects that store data on Filecoin are entering the Web3 ecosystem. Although Web3 is still relatively in its nascent stage, many people believe that it provides inherent advantages compared to today's Web, and now is a good time to test and obtain first-mover advantages. Slingshot hosted a panel of experts in the Web3 ecosystem, and they discussed how to consider user adoption and how to expand the market for Web3 products.
Members of this group include Shi Khai Wei, COO of Longhash Ventures, Gabriel Anderson, MD of Tachyon Accelerator, and Colin Evran, head of the Filecoin ecosystem. Pooja Shah, Head of Filecoin Product, moderated the panel discussion.
Web3 is a movement in the field of software development, and applications are being transformed from building based on a centralized infrastructure to a decentralized protocol. Applications built on the Web3 architecture do not have to have a single point of control, but can eliminate intermediaries, which brings many benefits. Filecoin is the key to the Web3 ecosystem, it allows files to be stored and retrieved from anyone willing to put their hardware on the network.
The Web3 ecosystem has come a long way, and there are many promising projects in this area. From developing cool technologies just for technology to building actual products for users, our team members encourage potential builders to start thinking from a more market-oriented perspective.
Some insights from group members:
Gabriel Anderson (Gabriel Anderson): "We are in the early days of the Web3 life cycle, but now certain applications are beginning to rise, and we are seeing some really innovative early use cases. For example, whether you know it or not, everyone has Digital assets, and there is a problem with inheriting the assets. A company is creating a seamless experience for people to deliver their digital assets. There is also a complete market for video transcription services, which can solve the delay problem at a fraction of the price."
Colin Evran: "We shouldn't ignore the strength of the early adopter spirit in the Web3 space. The kind of innovation catalyzed by these early adopters working in the same space is incredible."
Shi Kai: "As developers, we need to focus on building real users and communities that will be with us. Looking for real user communities that can provide you with real feedback, not people who think it is cool just because of new things.