Hello everyone, I am Dugufeng. In 10 years, I transformed from a port worker to a state-owned enterprise's big data manager through self-study. And register the big data mobile public account, and continue to create articles. I am very happy to meet you here~
I believe everyone in yesterday's circle of friends was overwhelmed by this exciting news.
Form a National Data Agency!
Video number related news arrangement
After so many years of development, big data has been widely used in many fields. And it has gradually become an infrastructure, just like when the Internet was just emerging. Data has become an asset, affecting economic development and everyone's life.
The top priority of the National Data Administration is to promote the development of the digital economy, organize the implementation of the national big data strategy, promote the construction of the basic system of data elements, and promote the layout and construction of digital infrastructure, etc.
The establishment of the National Data Bureau is just the beginning, and there will be more good news in the future. So, what trends will this great change bring about?
What are the opportunities for us data practitioners and friends who want to switch to big data?
1. Further expansion of big data talent gap
Some people will say that there are more and more programmers now, universities are constantly training them, and Internet-related jobs have become more and more saturated. Some technical positions may be saturated, but there are always gaps in data positions.
The development of big data has gone through a process from expansion to stability. Around 2012, Hadoop-related big data technology began to rise, bringing about huge technological innovations. However, many companies regard big data more as a gimmick. The purpose of making big data is to make a cool big screen display, but the amount of data collected and processed is actually not large. As a result, a large amount of big data work is still done by background developers and front-end developers. A large number of big data practitioners have no chance to come into contact with large data volumes. Without the baptism of large data volumes, it is impossible to understand the essence of big data.
In the next few years, concepts such as big data platforms and data platforms emerged in an endless stream. Many companies have no experience in big data, dig cars out of thin air, and then fool other companies to achieve profit goals. But as time goes by, big waves rush for gold, and after the waves pass, you will know who is swimming naked~
Therefore, in enterprises, talents who can cope with the challenge of large data volume, establish a big data architecture, and understand the essence of big data are still very rare. Around 2017, some colleges and universities established big data-related majors such as big data science and technology , and began to train big data professionals. There are not many, and there are still very few talents that can be provided for enterprises.
With the further implementation of big data, the gap of big data professionals will only become bigger and bigger. I believe that more people will switch to the big data industry, but everyone knows my experience. The road to change is very difficult. Difficulties, opportunities and challenges will coexist~
2. The demand for positions related to data governance has surged
The lack of management of data will cause a series of problems, such as poor data quality, data security issues and so on. Moreover, many companies are eager for quick success and instant benefit, trying to find one or two big data R&D companies to complete the big data system, but in fact they have dug a big hole for themselves, which may not necessarily produce any results, but will leave hidden dangers.
In recent years, the state has formulated a series of laws and regulations to restrict the use of data by enterprises. Last year, huge fines were issued to non-compliant enterprises. The establishment of the National Data Bureau will make relevant supervision work more effective. , especially the financial industry, and industries involving personal information security, should pay more attention to the use of their own data.
For data practitioners, we must also be aware of the trend, and the general direction of data governance is irreversible. At present, the proportion of big data technicians is much larger than that of data governance personnel, but this proportion will continue to shrink. Mastering the knowledge of data governance will not only make the daily work of big data more organized, but also bring more benefits to oneself. Many job opportunities.
At present, national-level data governance certification has not yet been issued, but international data governance-related certification and knowledge can still be learned, which will also be of great help to future certification.
3. Acceleration of big data and AI integration
The explosion of ChatGPT this time was unexpected by many people, and a large number of people stood up and said that the era of artificial intelligence is coming.
But those who understand the underlying principles should know that the reason why ChatGPT is so easy to use and can answer any questions is rooted in the accumulation of large amounts of data. The training of the model with large amounts of data makes it more and more intelligent, and it is almost useless. I don't know.
For quite a long time, everyone believed that big data has little to do with artificial intelligence. Big data does data ETL every day, and artificial intelligence uses Python to write algorithms. But when it really needs to be implemented, it is discovered that big data and artificial intelligence must be integrated. In recent years, many companies have established data intelligence departments. Big data and artificial intelligence have gradually come together. Artificial intelligence without the support of big data is difficult to implement, and it is difficult for a smart woman to cook without rice.
4. Domestic open source data software will shine
As soon as the news came out yesterday, many people have already searched for big data-related stocks, ready to start a new round of hype. Of course, the A-share market is also certain, and we will not comment on it.
But for commercial software in the field of big data, we have to admit that there is a gap between us and foreign countries. The atmosphere of open source technology in foreign countries is stronger, and a lot of excellent software has been born. Including my recent research on metadata management tools, data visualization tools, etc., are some of the leading foreign countries.
But it is inevitable to rely on foreign countries all the time, but we are particularly excited to see that some domestic open source projects have begun to shine and have become top Apache projects. For example, the open source scheduling tool DolphinScheduler, the stream processing development framework StreamPark, etc.
These excellent open source projects, and the outstanding big data developers behind them, are the future of big data in China!
I'm Dugufeng, if you like my article, I hope you can forward it, like it, watch it and support me, see you in the next article!
Recommendation of Popular Articles on Big Data Flow
Big Data Data Governance | WeChat Exchange Group~
What exactly is CDMP - a super-comprehensive introduction to the international certification of data governance
Open Source Data Quality Solutions - Apache Griffin Getting Started
One-stop Metadata Governance Platform - Datahub Getting Started
Pre-research on data quality management tools - Griffin VS Deequ VS Great expectations VS Qualitis
Thousand-character long text - Datahub offline installation manual
Metadata Management Platform Datahub2022 Annual Review
Big data flow: big data, real-time computing, data governance, and data visualization practice self-media. Regularly publish data governance and metadata management implementation technology practice articles, and share relevant technologies and materials for data governance implementation implementation.
Provide learning exchange groups such as big data introduction, data governance, Superset, Atlas, Datahub, etc.
Big data flows, and the learning of big data technology will never stop.