The field of big data is very broad. Whether it is the technology field, the food industry, or the retail industry, big data talents are needed to process big data to provide better user experience, optimize inventory and reduce cost forecasting needs.
What does big data development do?
There are two types of big data development, writing Hadoop and Spark applications and developing the big data processing system itself. The big data development engineer is mainly responsible for the development and maintenance of the company's big data platform, architecture design and product development of related tool platforms, network log big data analysis, real-time computing and stream computing, data visualization and other technology research and development and network security business theme construction. model work.
Skills required for big data development:
The languages currently engaged in the development of big data applications include Java, Python, Scala, R, etc. It is necessary to be familiar with the principles and usage methods of Hadoop, HBbase, hive, spark, Flink, ES, Presto, Flume, and Kafka ecology, and master data development and data mining of various processes.
If you want to meet the company's employment standards, education, work experience, and mastery of skills are all very important~
Let’s first look at the report data of several recruitment websites:
-
Released by Boss Zhipin, this spring’s recruitment data big data demand growth ranks second
-
Liepin released the five fields with the fastest year-on-year growth in new jobs since 2019. The top five are: artificial intelligence, manufacturing, big data, medical care, and energy and environmental protection.
-
The "2020 White Paper on the Development of China's Big Data Industry" shows that in 2019, the scale of China's big data industry reached 539.7 billion yuan, a year-on-year increase of 23.1%, and then grew steadily. It is expected to exceed one trillion yuan by 2022.
-
According to the statistical results of LinkedIn, CCID Think Tank, Lagou.com and other institutions, the overall gap of data talents in the era of big data is showing a growing state of intensification. In the past three years, the data talent gap has been increasing by 500,000 people per year. It is estimated that in 2022, after college graduates majoring in big data enter the job market on a large scale, the growth rate of the overall gap will slow down, but this gap is still will exist for a long time.
Recruitment is available, but applicants often encounter various problems in finding a job because of their academic qualifications and work experience. So what is the specific situation of developers who have been engaged in big data now? Let's look at the following aspects:
1. Academic level
From the perspective of education level, the education level of my country's big data talents is divided into 4 categories, namely master's degree and above, bachelor's degree, junior college, and junior college, among which the big data talents with bachelor's degree are the most, accounting for up to 65.45%. Followed by master's degree and above, and big data talents with junior college degree and below account for only a small part. It can be seen that the big data industry, as an emerging industry, generally has relatively high educational requirements for talents.
2. Professional sources
In terms of professional sources, the professional sources of big data talents in my country are mainly composed of four major categories: mathematics and science, economic management, computer and other majors, of which computer science accounts for the highest proportion, followed by mathematics and science.
3. Channel source
The channel sources of big data talents are divided into four categories, namely school recruitment, social recruitment, internal training and recommendation, and training institution recruitment. See the figure below for the number and proportion of the sources of big data talents in enterprises.
Among them, social recruitment accounts for the largest proportion, which is higher than the sum of school recruitment, internal training and promotion, and training institution recruitment. At present, it mainly relies on social recruitment, which shows that school education is out of touch with social needs, and internal training and training cannot meet job requirements.
4. Salary level distribution
At present, the salary of big data talents is at a relatively high level. Salaries below 10,000 yuan accounted for 34.6% of the total; 10,000 to 20,000 yuan accounted for 35.64%; and above 20,000 yuan accounted for 29.77%.
5. Type and number of posts
At present, the big data positions provided by enterprises can be divided into the following categories according to the job content requirements:
① Primary analysis category, including business data analysts, business data analysts, etc.
② Mining algorithms, including data mining engineers, machine learning engineers, deep learning engineers, algorithm engineers, AI engineers, data scientists, etc.
③ Development and maintenance, including big data development engineers, big data architecture engineers, big data operation and maintenance engineers, data visualization engineers, data acquisition engineers, database administrators, etc.
④ Product operation category, including data operation manager, data product manager, data project manager, big data sales, etc. The number and proportion of the four types of posts are shown in the figure below.
The demand for big data is increasing, and the country is also opening related jobs, which have increased year by year since 2018.
At this time, students and parents who apply for university are also very interested in big data and artificial intelligence. Big data has entered the top 5 for three consecutive years, and a bachelor's degree is all that is required.
In the foreseeable next few years, this is really a sunrise industry, and there is a big gap now.
So if you want to know what kind of job you can find in the future and the salary of the job, let us show it in the form of data~
Then open Boss direct employment, search for big data engineers:
let's do data analysis:
The salary column has a minimum salary and a maximum salary. We compared and analyzed different cities and found that Beijing has the highest salary level, with the lowest being 22k and the highest being 38k.
Working years are also a big factor that restricts salary levels. It can be seen from the figure that even if you have just graduated, you can reach a salary range of 11-20k.
As far as educational requirements are concerned, most of them are undergraduates, followed by junior colleges and masters, and others are so few that they are not shown in the figure. 
Most of the requirements of enterprises for different positions are 3-5 years. Of course, enterprises need employees with certain work experience, but in actual recruitment, if you have project experience and no problem with theoretical knowledge, enterprises will relax the conditions.
Analyzing different industries, we found that the demand for big data jobs is distributed in all walks of life, mainly in computer software and the Internet, and it may also be determined by this recruitment software. After all, Boss direct employment is still mainly in the Internet industry.
Let's take a look at which companies are recruiting for big data-related positions. Judging from the number of more than 15, Huawei, Tencent, Ali, Byte, these big companies still have a large demand for this position.
So what skills do these jobs require? Spark, Hadoop, Data Warehouse, Python, SQL, Mapreduce, Hbase, etc.
According to the domestic development situation, the future development prospects of big data will be very good. Since enterprises have started digital transformation in 2018, first- and second-tier cities have a very strong demand for talents in the field of big data. In the next few years, the demand for talents in third- and fourth-tier cities will also increase significantly.
Big data learning route and resources:
Getting Started: Getting Started with Linux → MySQL Database
Core Foundation: Hadoop
Data Warehouse Technology: Hive Data Warehouse Project
PB Memory Computing: Getting Started with Python → Advanced Python → pyspark Framework → Hive+Spark Project
Before choosing a training institution, you can learn the basics of big data first to see if you can master it~
This set of tutorials covers everything that must be learned in big data
Hadoop, Hive, cloud platform practical projects
Let zero-based students get started in one stop
Straight-through big data core technology
This new set of big data tutorials is based on Hadoop, Hive, cloud platform and other technologies to lead you into the field of big data from shallow to deep, and experience the charm of large-scale data computing together.
Based on the content design of zero-based learning, it provides a wealth of supplementary knowledge points for zero-based students to carry out pre-learning.
As a new big data introductory course in 2023, the course content adopts a new technology stack system. Based on Hadoop3.3.4, Hive 3.1.3, Alibaba Cloud and UCloud cloud platforms, an introductory course for students to create a big data Hadoop ecosystem, but not just Hadoop.
course features
• Perfect combination of theory + practice: This set of tutorials uses the form of "theory + practice" to comprehensively introduce the relevant knowledge of big data Hadoop and Hive offline development;
• Both content and depth: the course adopts the content design of "introduction + improvement", the introductory knowledge and advanced knowledge are independent of each other, first comprehensive introduction, then comprehensive advanced, step by step so that everyone can learn something;
• Combining the current popular cloud platforms (Aliyun, UCloud) to bring you "Cloud Native Big Data Development": based on Hadoop3.3.4, Hive 3.1.3, Alibaba Cloud and UCloud cloud platforms, using a new technology stack system.
suitable for the crowd
>Basic zero: beginners to advanced level, and then to proficiency
＞Advanced: Experienced engineers consolidate and expand
＞Explorer: those interested in enjoying the charm of big data
Getting Started with Big Data Development in Phase 1
Pre-study guide: Start with traditional relational databases, master data migration tools, BI data visualization tools, and SQL, and lay a solid foundation for subsequent learning.
1. Big data data development foundation MySQL8.0 from entry to proficiency
MySQL is the entire IT basic course, and SQL runs through the entire IT life. As the saying goes, if SQL is well written, you can find a job easily. This course fully explains MySQL8.0 from zero to advanced level. After studying this course, you can have the SQL level required for basic development.
The core foundation of big data in the second stage
Pre-study guide: learn Linux, Hadoop, Hive, and master the basic technology of big data.
2022 Big Data Hadoop Introductory Tutorial
Hadoop offline is the core and cornerstone of the big data ecosystem, an introduction to the entire big data development, and a course that lays a solid foundation for the later Spark and Flink. After mastering the three parts of the course: Linux, Hadoop, and Hive, you can independently realize the development of visual reports for offline data analysis based on the data warehouse.
The third stage of hundreds of billions of data warehouse technology
Pre-study guide: The course at this stage is driven by real projects, learning offline data warehouse technology.
Data offline data warehouse, enterprise-level online education project practice (complete process of Hive data warehouse project)
This course will establish a group data warehouse, unify the group data center, and centralize the storage and processing of scattered business data; the purpose is from demand research, design, Version control, R&D, testing, and launch, covering the complete process of the project; digging and analyzing massive user behavior data, customizing multi-dimensional data sets, and forming a data mart for use in various scene themes.
The fourth stage PB memory computing
Pre-study guide: Spark has officially adopted Python as the first language on its homepage. In the update of version 3.2, it highlights the built-in bundled Pandas; Spark content.
1. From entry to mastery of python (19 days)
Python basic learning courses, from building the environment. Judgment statements, and then to basic data types, and then learn and master functions, familiarize yourself with file operations, initially build object-oriented programming ideas, and finally lead students into the palace of python programming with a case.
2. Python programming advanced from zero to website building
After completing this course, you will master advanced Python syntax, multi-tasking programming, and network programming.
3.spark3.2 from basic to proficient
Spark is the star product of the big data system. It is a high-performance distributed memory iterative computing framework that can handle massive amounts of data. This course is developed based on Python language learning Spark3.2. The explanation of the course focuses on integrating theory with practice, which is efficient, fast, and easy to understand, so that beginners can quickly master it. Let experienced engineers also gain something.
4. Big data Hive+Spark offline data warehouse industrial project actual combat
Through the big data technology architecture, it solves the data storage and analysis, visualization, and personalized recommendation problems in the industrial Internet of Things manufacturing industry. The one-stop manufacturing project is mainly based on the Hive data warehouse layer to store the data of various business indicators, and based on sparkSQL for data analysis. The core business involves operators, call centers, work orders, gas stations, and warehousing materials.