Resume
·Personal Information· ____________________________________________________________
Name: Gender:
Age: Work Experience:
Education: Undergraduate (full-time) Major: Computer Science and Technology
Place of Origin: English Level:
Telephone: E-mail:
·Job Intention· __________________________________________________________
Target Salary: Negotiable
job Position: Big Data Development Engineer
Job Status: Leaving
· IT Skills · ____________________________________________________________
1. Big Data Technology
1. Be able to build Hadoop and skillfully use related technologies HDFS, Yarn, MapReduce, Flume, Sqoop;
2. Master the principle of Mapreduce job scheduling and understand deeply Mapreduce operation principle and Shuffle process;
3. Familiar with Spark workflow, able to use Spark Sql for data processing and SparkStreaming for streaming computing;
4. Able to use message middleware Kafka for data caching;
5. Master the working principle of Zookeeper, and Can use Zookeeper to build HadoopHA cluster and SparkHA;
6. Familiar with full-text search Lucene and Elasticsearch;
7. Able to use reverse proxy Nginx to separate dynamic and static requests and load balance servers;
8. Understand Storm architecture and be able to use Storm for real-time computing.
2. Database related technologies
1. Be able to use Hive and Hbase to analyze and process massive data;
2. Be able to operate MySQL database for data storage;
3. Familiar with the use of MongoDB and Redis.
3. Programming language
1. Be able to use Java for programming;
2. Be able to use Scala for Spark operation;
3. Understand Shell script programming;
4. Understand the basic usage of Python.
4. Others
1. Be able to use Spring, SprngMVC, SpringBoot, MyBatis and other frameworks to build projects;
2. Familiar with common Linux commands;
3. Familiar with development tools such as GitHub and Maven.
·Working Experience· __________________________________________________________________________________
April 2017-March 2019
Company Industry: Computer Software Job Position: Big Data Development Engineer
Job Responsibilities:
1. Mainly responsible for distributed storage of big data;
2. Analysis of offline data sources, real-time data analysis Calculation;
3. Design of data acquisition, cleaning and other programs.
·Project Experience· ____________________________________________________________
Project 1:
Caomin Film Network Data Analysis Development Cycle August 2018-February 2019
Technical Implementation:
Hadoop+Flume+Kafka+Mycat+Hive+Spark+SparkMLlib+
Redis+MongoDB+Elaticaserch
Project Position: Big Data Development engineer
Project introduction:
This project is to conduct rating statistics for movie websites. Among the various TV series and movie programs watched by users, in order to facilitate us to know which TV series and movies are more popular with the audience, we can monitor through the viewing status of some users , Find all kinds of popular TV series and movies, and count the number of registered people every day in real time.
Project Responsibilities:
1. Responsible for participating in the offline statistical analysis of website indicators: such as statistics of the average score of movies, statistics of high-quality movies in each category, statistics of the most popular movies, statistics of high-quality movies, etc.; 2. Collect data, clean and store the data
in to Hdfs;
3. Use the collaborative filtering ALS algorithm in Spark MLlib to calculate user movie recommendation matrix and movie similarity matrix;
4. Use ES to calculate content-based recommendation results, etc.
Technical points:
1. Flume monitoring log data is transmitted to Kafka;
2. Hadoop and ES respectively pull data from Kafka and perform real-time cleaning and storage;
3. Use SparkSql to pull data from Hadoop for offline data analysis and calculation;
4 . Use Spark Streaming to pull data from Kafka for real-time calculation;
5. Use Spark MLlib's ALS recommendation algorithm to analyze and recommend offline videos to users;
6. Use the algorithm process provided by the company to calculate real-time recommended videos.
Project 2:
Game data indicator analysis development cycle April 2018-August 2018
Technical implementation:
Hadoop+Zookeeper+Sqoop+Mycat+Hive+Kafka+Flume+Spark+Redis+Nginx+Hbase
Project position: Big data development engineer
project Introduction:
Through the analysis of various indicators of game data, it can help game operators understand the behavior and needs of players, and through the feedback information of players, problems in the game can be continuously corrected, so that the game can be operated in a healthy, stable and sustainable manner. .
Project Responsibilities:
1. Statistics of active players: DAU, WAU, MAU, DAU/MAU, player level, region, age, gender distribution; 2.
Statistics of loyal players (7, 14, 30 continuous online)
3. Statistics of player retention: Retained players on the next day, week and month, player retention conditions, player level, number of games, whether to pay, etc.
4. Statistics of player loss (7, 14, 30 continuous offline), return players on the day, level of lost players before loss, number of games, whether to pay, etc.
Technical points:
1. Kafka obtains data from Nginx;
2. Use Flume to pull data from Kafka to Hdfs for cleaning and storage;
3. Use SparkSql to pull data from Hdfs for offline data calculation;
4. SparkStreaming pulls data from Kafka The data is calculated in real time.
Project 3:
Peripheral Tourist Flow Data Analysis Development Cycle November 2017-April 2018
Technical Implementation:
Hadoop+Flume+Kafka+Hive+Mycat+Spark+Redis
Project Position: Big Data Development Engineer
Project Introduction:
The passenger flow analysis system It is for the surrounding tourism websites to carry out passenger flow of scenic spots, environmental analysis of popular scenic spots, customer source market insight, and marketing theme analysis. The data obtained after analysis are displayed to the front end, so as to provide daily decision-making support for the website, such as some tourist attractions If the popularity of the site is high, more recommendations can be made for the attraction.
Project responsibilities:
data cleaning operation;
some calculations of offline indicators such as:
1. Annual passenger flow
statistics
of designated routes;
4. Market analysis of mature scenic spots - ranking of prefectures and provinces;
5. Analysis of income and exit ratio of mature scenic spots - ranking of provinces and cities;
6. Analysis of conversion rate of potential tourist source cities - ranking of prefectures and cities.
Technical points:
1. Use Flume to monitor logs and store them in Hdfs;
2. Use MR to clean and store data;
3. Use Sparksql to read data from the data warehouse and analyze the data;
4. Flume directly sends data to Kafka, and Sparkstream connects to Kafka Perform real-time calculations.
·Self-evaluation· ______________________________________________________________
1. Familiar with hadoop distributed storage, able to use self-written MR programs to solve problems;
2. Good learning, communication and organization skills;
3. Technically, have good independent module completion and problem-solving skills ;
4. Good adaptability, can withstand strong work pressure, and can quickly integrate into the team;
5. Sensitive to cutting-edge technologies, very willing to study big data-related technologies.