The experience of big data veteran drivers-take your own characteristic technical path

The experience of big data veteran drivers-take your own characteristic technical path

Dean Lang Langjian talks about big data

The main reason why Langjian wrote this article is that many questions asked by questioners in qq, WeChat groups, and the planet are caused by the wrong way of solving problems or thinking. There is also how to do when learning a new technology?

Many people are looking for videos from the top of the waves. I think this behavior is not very good, and old drivers would not do it. So, today Langjian will write an article about how Langjian learns a new technology or solves problems at work.

1. Official website

The first thing to look at the official website, many problems are caused by my own concepts, or do not understand the principles of using the framework.
For example, this kind of problem

Well, in fact, there is a detailed explanation at the bottom of the official website

http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html

This is to do things according to your own thinking inertia, not according to the official website manual. So, you have to look at the official website more. I think it is necessary to go through the official website three or five times when you are just starting to learn something.

Don't pretend that I don't know English, Google Translate is becoming more and more accurate.

2. Case

When many people learn big data, they don't know how to write some demos by hand, and they always want to rely on videos to gain practical experience in projects. You know, the so-called 100 billion-level data projects in the video are all demos. No one will give you hundreds of billions of data for you to practice. Of course, even if you give it to you, you can't play it, and you have no resources.

I emphasize here that the project is not the focus. The focus is on the principle of engineering understanding. Only when you understand the principle can you easily locate the bottleneck and find an optimization method.

There are still many Xiaobai looking for demos, which is actually unnecessary. First, you can go to github to search, and secondly, you can download the source code and see the demo in the source code. There are many cases in the examples module in the general source code.

For example, the demo case in the spark source code,

Another example is the demo case in the flink source code,

There are many cases, and it will support multiple programming languages.

These are enough for you to get started

3. The definitive guide

The authoritative guide is not necessarily called this name, it is a general term for books with better knowledge. But for the big data field, a lot of good information is really called the authoritative guide, if you don't have it, just go to the official website. Such as hbase authoritative guide, hadoop authoritative guide. I think whether you are a beginner or an old driver, in addition to the official website, you have to gnaw a few times and the authoritative guide to enrich your brain. Otherwise, there is no relevant content in the brain. Then coding is not blind or according to thinking. Is it inertia? Let alone tuning.

Operate HBase under Spark (1.0.0 new API)

There are also various interview questions of hadoop. You must read the authoritative guide of hadoop. Many blogs are copied from the authoritative guide of hadoop. There are always clerical errors in people copying. I don’t understand. Why would you rather look at content that may have clerical errors? Why not read the authoritative guide? The official website is similar.

4. Search Engine

Google Baidu.

There is no need to say more about keyword search. Tucao, there are too many ads on Baidu, I suggest you get Google.

5. Ask Daniel

Just ask others. The question asked should be described clearly, as well as the solution you tried, so that it is convenient for others to solve the problem for you in a targeted manner and filter out unnecessary guesses.

Taboo, right?

Taboo, some people will. . . ?

.....

Everyone is very busy, so when asking others questions, be sure to go straight to the topic, you can refer to:

1) Post a detailed log screenshot

Don’t just post one line or half line, the big cow is not a god;

Don't send the error log directly. It is not easy to observe and read. Just send a screenshot to get as long as possible.

2) Write the background of the problem

For example, when you are doing something, which line of code is pointed out by the log, it is best to post it.

3) Have you tried to solve it yourself, and what methods have you adopted?

4) Good attitude.

Regardless of whether someone is willing to help you, you will have a clearer mind when sorting out this set of problems, and maybe you will find a solution.

6. Source code

There are everything you want in the source code, so I won't talk about it here. I will post an article on how to use the source code in detail.

The general policy for the source code is:

Don't be afraid of the source code, extract and modify the source code according to the needs of the work.

The above is the experience summarized by Langjian. It is necessary to deal with the actual situation flexibly and form your own method instead of just copying everything.