This is an industry that relies on technology to make a living. As long as you have the ability, who will discriminate against you? If you have no technology or ability, both men and women will be complained to a certain extent~
In the past, I always heard people say that girls are not suitable for studying IT, that they get bald quickly when they do IT, and that they work overtime seriously. In fact, these are alarmist words. Both boys and girls can be programmers, and as long as they like it, they can do it well through hard work.
Moreover, in the IT technology industry, girls still have a great advantage in learning big data. Compared with boys, girls are more patient, tolerant, delicate, and have a better grasp of users' psychology. It is more aesthetically pleasing and has better control over the page details. It can also help the team balance men and women and activate the atmosphere.
Big data is a profession that will pay off as long as you are willing to learn. Regardless of whether you are a boy or a girl, regardless of your height or academic qualifications, as long as your skills reach the required level, you can get a high-paying job.
In the Internet era, as the tide of the Internet goes to the bottom, and traditional enterprises are undergoing digital transformation one after another, basically every company is considering how to further tap the value of data and improve the operational efficiency of enterprises. In this trend, big data technology is becoming more and more important. Therefore, in the future, big data is one of the necessary skills for our workers . Friends who want to make a difference in the Internet field must seize the wave of rapid Internet development. Go forward boldly toward the goal in your heart!
The technical requirements of a big data engineer are as follows:
1. Master at least one database development technology: Oracle, Teradata, DB2, Mysql, etc., and flexibly use SQL to realize ETL processing of massive data; 2. Familiar with
Linux system conventional shell processing commands, and flexibly use shell to do text processing and system operations;
3. Experience in distributed data storage and computing platform application development, familiarity with Hadoop eco-related technologies and relevant practical experience are preferred, focusing on Hdfs, Mapreduce, Hive, Hbase; 4. Proficiency in one or more programming languages
, Those who have experience in large-scale project construction are preferred, focusing on Java, Python, and Perl;
5. Those who are familiar with the knowledge and skills in the field of data warehouse are preferred, including but not limited to: Data management;
6. Master real-time streaming computing technology, experience in storm development is preferred.
Data engineers aim to look at the big picture and develop. Data engineers build automated systems and model data structures so that data can be processed efficiently. The goal of a data engineer is to create and develop tables and data pipelines to support analytics dashboards and other data customers (such as data scientists, analysts, and other engineers). Much like most engineers, there are many designs, assumptions, constraints, and developments to be able to create some sort of ultimate robust system. This system might be a data warehouse and ETL or streaming pipeline.
Big data learning route and resources:
Introduction to development: Introduction to Linux → MySQL database
Core foundation: Hadoop
data warehouse technology: Hive data warehouse project
PB memory calculation: Introduction to Python → Advanced Python → pyspark framework → Hive+Spark project
Getting Started with Big Data Development in Phase 1
Pre-study guide: Start with traditional relational databases, master data migration tools, BI data visualization tools, and SQL, and lay a solid foundation for subsequent learning.
1. Big data data development foundation MySQL8.0 from entry to proficiency
MySQL is the entire IT basic course, and SQL runs through the entire IT life. As the saying goes, if SQL is well written, you can find a job easily. This course fully explains MySQL8.0 from zero to advanced level. After studying this course, you can have the SQL level required for basic development.
The core foundation of big data in the second stage
Pre-study guide: learn Linux, Hadoop, Hive, and master the basic technology of big data.
2022 Big Data Hadoop Introductory Tutorial
Hadoop offline is the core and cornerstone of the big data ecosystem, an introduction to the entire big data development, and a course that lays a solid foundation for the later Spark and Flink. After mastering the three parts of the course: Linux, Hadoop, and Hive, you can independently realize the development of visual reports for offline data analysis based on the data warehouse.
The third stage of hundreds of billions of data warehouse technology
Pre-study guide: The course at this stage is driven by real projects, learning offline data warehouse technology.
Data offline data warehouse, enterprise-level online education project practice (complete process of Hive data warehouse project)
This course will establish a group data warehouse, unify the group data center, and centralize the storage and processing of scattered business data; the purpose is from demand research, design, Version control, R&D, testing, and launch, covering the complete process of the project; digging and analyzing massive user behavior data, customizing multi-dimensional data sets, and forming a data mart for use in various scene themes.
The fourth stage PB memory computing
Pre-study guide: Spark has officially adopted Python as the first language on its homepage. In the update of version 3.2, it highlights the built-in bundled Pandas; Spark content.
1. From entry to mastery of python (19 days)
Python basic learning courses, from building the environment. Judgment statements, and then to the basic data types, and then learn and master the functions, familiarize yourself with file operations, initially build an object-oriented programming idea, and finally lead students into the palace of python programming with a case.
2. Python programming advanced from zero to website building
After completing this course, you will master advanced Python syntax, multi-tasking programming, and network programming.
3.spark3.2 from basic to proficient
Spark is the star product of the big data system. It is a high-performance distributed memory iterative computing framework that can handle massive amounts of data. This course is developed based on Python language learning Spark3.2. The explanation of the course focuses on integrating theory with practice, which is efficient, fast, and easy to understand, so that beginners can quickly master it. Let experienced engineers also gain something.
4. Big data Hive+Spark offline data warehouse industrial project actual combat
Through the big data technology architecture, it solves the data storage and analysis, visualization, and personalized recommendation problems in the industrial Internet of Things manufacturing industry. The one-stop manufacturing project is mainly based on the Hive data warehouse layer to store the data of various business indicators, and based on sparkSQL for data analysis. The core business involves operators, call centers, work orders, gas stations, and warehousing materials.