Scientific progress is increasingly driven by data. Massive data brings both opportunities and new challenges to data analysis. Big data is often obtained by using many technologies and methods to synthesize information from multiple channels and at different times. What are the core principles of big data technology?
Data is value is an extremely respected concept in the computer field. No matter how much data is, it is attributed to big data. Data analysis is becoming more and more popular, and capital is also rushing to companies with big data labels. It has been repeatedly evaluated and pursued just like a mobile digital currency. Data can tell us the consumption tendency of each customer, what they want, what they like, what are the differences between each person's needs, and which can be grouped together for classification.
The principle of data core: from the core of "process" to the core of "data"
In the era of big data, the computing model has also changed, from the core of "process" to the core of "data". The distributed computing framework of the Hadoop system is already a paradigm with "data" as its core. Unstructured data and analysis requirements will change the way IT systems are upgraded: from simple increments to structural changes. The new thinking under big data-the transformation of computing mode.
Scientific progress is increasingly driven by data. Massive data brings both opportunities and new challenges to data analysis. Big data is often obtained by using many technologies and methods to synthesize information from multiple channels and at different times. In order to meet the challenges brought by big data, we need new statistical ideas and calculation methods.
Data value principle: Function is value, and data is value
What is really interesting about big data is that data has become online, which is precisely the characteristic of the Internet. The function of the product in the non-Internet era must be its value, and the data of today's Internet product must be its value.
Data can tell us the consumption tendency of each customer, what they want, what they like, what are the differences between each person's needs, and which can be grouped together for classification. Big data is an increase in the amount of data, so that we can realize the process from quantitative change to qualitative change.
The principle of full sample : change from sampling to requiring all data samples
All data samples are needed instead of sampling. The things you don’t know are more important than the things you know, but if there are enough data now, it will make people see and feel the rules.
The data is so large and so many, so people feel that they have enough ability to grasp the future, a kind of judgment about the uncertain state, and then make their own decisions. These things sound very primitive to us, but the thinking behind it is actually very similar to the big data we are talking about today.
The principle of focusing on efficiency: changing from focusing on accuracy to focusing on efficiency
Focusing on efficiency rather than accuracy, big data marks a big step forward for human beings in seeking to quantify and understand the world. Many things that were not measurable, stored, analyzed, and shared in the past have been digitized, with a large amount of data and More less precise data opens a new door for us to understand the world. Big data can improve production efficiency and sales efficiency. The reason is that big data can let us know the needs of the market and people's consumption needs. Big data makes the decision-making of enterprises more scientific, changing from focusing on accuracy to improving efficiency. Big data analysis can improve the efficiency of enterprises.
Competition is the driving force of an enterprise, and efficiency is the life of an enterprise. Low efficiency and high efficiency are the key to measuring the success or failure of an enterprise. Generally speaking, the ratio of input to output is efficiency, and the pursuit of high efficiency is the pursuit of high value. The efficiency is different between manual, machine, automatic machine, and intelligent machine. Intelligent machine is more efficient and can replace human thinking labor. The core of smart machines is big data braking, and big data braking is faster. In a rapidly changing market, rapid prediction, rapid decision-making, rapid innovation, rapid customization, rapid production, and rapid listing have become the norms of corporate actions. That is to say, speed is value, efficiency is value, and all these are inseparable from big data thinking. .
The principle of focusing on correlation: change from causality to focusing on correlation
Focusing on correlation instead of causality, society needs to give up its desire for causality and only need to focus on correlation, which means that it only needs to know what it is, not why. This overturns the conventions since ancient times, and our most basic way of making decisions and understanding reality will also be challenged.
In this uncertain era, when we find the exact cause and effect relationship and then do things, this matter is no longer worth doing. Therefore, the thinking in the "big data" era is a bit like the mechanical thinking that has returned to the industrial society-mechanical thinking means that when you press that button, the corresponding result will definitely appear. This is the state. As agricultural society moves forward, it does not need to find a very close and clear causal relationship, but only need to find the relevant relationship, only need to find the signs. The society therefore gave up the traditional preference of looking for causal relationships and began to explore the benefits of related relationships.
Buildings that are illegally partitioned inside the house are much more likely to catch fire than other buildings. New York City receives 25,000 complaints about overcrowded housing every year, but there are only 200 inspectors in the city who handle complaints. A team of analysts in the Mayor’s Office feels that big data can help solve this gap in demand and resources. The team established a database of all 900,000 buildings in the city, and added data collected by 19 departments in the city: tax arrears seizure records, abnormal use of water and electricity, payment arrears, service cutoffs, ambulance use, Local crime rates, rat complaints, and so on.
Next, they compared this database with building fire records ranked by severity in the past 5 years, hoping to find correlation. Sure enough, the type of building and the year of construction are factors related to the fire. However, an unexpected result is that there is a correlation between buildings that have obtained external brick wall construction permits and a lower incidence of serious fires. Using all this data, the team established a system that can help them determine which housing congestion complaints require urgent handling. The various characteristic data of the buildings recorded by them are not the cause of the fire, but these data are related to the increase or decrease of fire hazards. This knowledge has proved to be extremely valuable: in the past, only 13% of house vacancies were issued by house inspectors when they appeared on the scene. After the adoption of the new method, this proportion has risen to 70%-efficiency has been greatly improved.
Business people all over the world are chanting the advantages of the advent of the big data era: how a supermarket discovered the fact that a 17-year-old girl was pregnant from the shopping list of a 17-year-old girl; or put beer and diapers together for sale, it’s amazing Increased the sales of both parties. The information revealed by big data does sometimes subvert. For example, a statistics from Tencent on social networks shows that more than twice as many men love to watch family dramas than women; Chinese aunts care about the price of gold the most, but the post-90s are closely followed. In the past year, all of the top ten wireless payment ratios in Alipay were in Qinghai, Tibet and Inner Mongolia.
Forecasting principle: from unpredictable to predictable
The core of big data is prediction, and big data can predict in many ways. Big data is not about teaching machines to think like humans, on the contrary, it is about applying mathematical algorithms to massive amounts of data to predict the possibility of things happening. Precisely because in the face of the laws of big data, everyone's behavior is the same as everyone else's, with no essential changes, so businesses will change consumer behaviors more than consumers.
In addition, as the system receives more and more data, the system can be improved by recording the best predictions and patterns found. It is often seen as part of artificial intelligence, or more precisely, as a type of machine learning. The real revolution is not in the machines that analyze data, but in the data itself and how we use it. Once statistics and the current large-scale data are integrated, it will subvert a lot of our original thinking. So now there are more and more things that can become data, and the ability to calculate and process data is getting stronger, so everyone suddenly found this thing very interesting. So, what can big data do? Can do a lot of interesting things.
The Internet, mobile Internet and cloud computers ensure the possibility of big data real-time forecasting, and also provide enterprises and users with real-time forecasting information, and related forecasting information, allowing enterprises and users to seize the opportunity. Because of the full sample nature of big data, people and people are the same, so the efficiency and accuracy of cloud computer software forecasts are greatly improved. There will be this kind of result if there are signs of this kind.
The principle of information searching for people: from people searching for information to information searching for people
The development of the Internet and big data is a process from people looking for information to information looking for people. First, people looked for information, people looked for people, and information looked for information. Now is the era of information looking for people. The era of information seeking people means that, on the one hand, we have returned to a kind of original. The broadcasting mode is information seeking people. We listen to the radio and we watch TV. It is information pushed to us, but there is a flaw. I don’t know what we are. Who, later, the Internet did the opposite, providing search engine technology to let me know how to find the information I needed, so search engine is a very critical technology.
Big data has also changed information superiority. According to evidence-based medicine, the first thing to treat a disease now is not to study pathology, but to use past data to study how to treat under the same circumstances. This leads to the loss of the information advantage between experts and ordinary people. I used to believe in doctors because they know a lot, but now I can check on Google to find out what disease I have.
Google has a machine translation team. At the beginning, the translated text could not be understood at all, but now 60% of the content can be read. There is a joke in the Google machine translation team, saying that every time a linguist leaves the team, the quality of translation will improve. The more the expert, the more confused it is, but breaking the convention and letting the data speak, the faster the speed of obtaining the truth.
The principle that machines understand people: From people understand machines to machines understand people better
Not to make people understand the machine better, but to make the machine understand people better, or to be able to use the machine even when the user is stupid. It’s not even for people to understand the environment, but for our environment to understand us, and the environment to adapt to people. To a certain extent, the natural environment cannot say that, but in the digital environment, there is already such a trend, that is, the world in which we live. , It tends to be more suitable for us and understand us better. Which company can truly make machines understand people better, make the environment understand people better, and let the whole world of life we carry with us understand us better, then it must be competitive, and "big data" technology can help We can help.
One of the core goals of big data technology is to dig out the hidden laws from the huge and varied structure of the data, so as to maximize the value of the data. Computers replace people to mine information and acquire knowledge. The ability to quickly obtain valuable information from a variety of data (including structured, semi-structured, and unstructured data) is big data technology. In big data machine analysis, techniques such as semi-supervised learning, ensemble learning, and probabilistic models are particularly important.
Principles of e-commerce intelligence: Big data has changed the e-commerce model and made e-commerce smarter
Business intelligence has been redefined in today's era of big data. For example: After traditional enterprises enter the Internet, after mastering the application of "big data" technology, they will find a sense of sudden enlightenment. It is like looking for something in a dark room and can't find it. Suddenly they encounter a switch and find that it is so laborious. It turned out to be easy to find things. Big data thinking, in fact, is not a full name judgment, but a description of a certain latitude of our era.
The era of big data does not mean that our era has nothing but big data. Even in the Internet and IT fields, it is not everything. It just means that we have added such a clear light to the characteristics of our era, which leads us to think about the past. The state of existence, and a differentiated expression of our personal life state.
Of course, the same technology can also be used to diagnose diseases, recommend treatment measures, and even identify potential criminals. In other words, if you don’t know it yet, the medical examination company or the hospital reminds you to go for the examination as soon as possible. You may get some diseases. The business knows yourself better than you, and people like you will appear under certain circumstances. The possible changes. Just as the Internet has changed the world by adding communication functions to computers, big data will also change the most important aspects of our lives because it creates unprecedented quantifiable dimensions for our lives.
The principle of customized products: from the production of products by enterprises to customized products by customers
The next wave of reforms is mass customization, customizing products and services for a large number of customers, with low cost and personalization. For example, a consumer hopes that the car he buys is red and green, and the manufacturer is able to meet the requirements, but the price is not as unaffordable as handmade. Therefore, under the premise that manufacturers can afford the high costs brought by mass customization, to truly achieve personalized products and services, they must have a good understanding of customer needs, and behind this, they need to rely on big data technology.
In the era of Internet big data, merchants may finally be able to make precise price discrimination against each customer. Many of our current behaviors are relatively extensive. Airlines will give us mileage cards to accumulate mileage based on the number of kilometers flown, but in fact, different mileages flown by different customers have different contributions to the airline's profit. So one day a customer may receive a letter, "Congratulations, sir, you have been selected as a lucky customer, we have upgraded you to a platinum card in advance." This shows that this customer has contributed enough to the airline. . One day the bank said "Congratulations, your limit has been increased again," which means that too much money has been spent.
Precisely because of the laws of big data, everyone behaves like everyone else, without essential changes. Therefore, the business will be more of the consumer's behavior than the consumer. Maybe you are thinking, after a year of hard work, where do you want to go on holiday? Open e-Mail, and there will be mails from airlines and travel agencies.
Enterprise products are sold directly to users, eliminating the need for intermediary circulation, so that the price of the product can be sold at the ex-factory price, and if the sales fee wants to systematically learn big data , you can join the big data technology learning exchange deduction : 522189307 , welcome Add, understand that course introducers have gained benefits, online products have become users’ belief that cheap, and an online shopping market has been formed. In order for users to become fans of your products, you must understand their needs. Customized products become the wishes of users and become a new direction for enterprise development.
Scientific progress is increasingly driven by data. Massive data brings both opportunities and new challenges to data analysis. Big data is often obtained by using many technologies and methods to synthesize information from multiple channels and at different times. What are the core principles of big data technology?
Data is value is an extremely respected concept in the computer field. No matter how much data is, it is attributed to big data. Data analysis is becoming more and more popular, and capital is also rushing to companies with big data labels. It has been repeatedly evaluated and pursued just like a mobile digital currency. Data can tell us the consumption tendency of each customer, what they want, what they like, what are the differences between each person's needs, and which can be grouped together for classification.
The principle of data core: from the core of "process" to the core of "data"
In the era of big data, the computing model has also changed, from the core of "process" to the core of "data". The distributed computing framework of the Hadoop system is already a paradigm with "data" as its core. Unstructured data and analysis requirements will change the way IT systems are upgraded: from simple increments to structural changes. The new thinking under big data-the transformation of computing mode.
Scientific progress is increasingly driven by data. Massive data brings both opportunities and new challenges to data analysis. Big data is often obtained by using many technologies and methods to synthesize information from multiple channels and at different times. In order to meet the challenges brought by big data, we need new statistical ideas and calculation methods.
Data value principle: Function is value, and data is value
What is really interesting about big data is that data has become online, which is precisely the characteristic of the Internet. The function of the product in the non-Internet era must be its value, and the data of today's Internet product must be its value.
Data can tell us the consumption tendency of each customer, what they want, what they like, what are the differences between each person's needs, and which can be grouped together for classification. Big data is an increase in the amount of data, so that we can realize the process from quantitative change to qualitative change.
The principle of full sample : change from sampling to requiring all data samples
All data samples are needed instead of sampling. The things you don’t know are more important than the things you know, but if there are enough data now, it will make people see and feel the rules.
The data is so large and so many, so people feel that they have enough ability to grasp the future, a kind of judgment about the uncertain state, and then make their own decisions. These things sound very primitive to us, but the thinking behind it is actually very similar to the big data we are talking about today.
The principle of focusing on efficiency: changing from focusing on accuracy to focusing on efficiency
Focusing on efficiency rather than accuracy, big data marks a big step forward for human beings in seeking to quantify and understand the world. Many things that were not measurable, stored, analyzed, and shared in the past have been digitized, with a large amount of data and More less precise data opens a new door for us to understand the world. Big data can improve production efficiency and sales efficiency. The reason is that big data can let us know the needs of the market and people's consumption needs. Big data makes the decision-making of enterprises more scientific, changing from focusing on accuracy to improving efficiency. Big data analysis can improve the efficiency of enterprises.
Competition is the driving force of an enterprise, and efficiency is the life of an enterprise. Low efficiency and high efficiency are the key to measuring the success or failure of an enterprise. Generally speaking, the ratio of input to output is efficiency, and the pursuit of high efficiency is the pursuit of high value. The efficiency is different between manual, machine, automatic machine, and intelligent machine. Intelligent machine is more efficient and can replace human thinking labor. The core of smart machines is big data braking, and big data braking is faster. In a rapidly changing market, rapid prediction, rapid decision-making, rapid innovation, rapid customization, rapid production, and rapid listing have become the norms of corporate actions. That is to say, speed is value, efficiency is value, and all these are inseparable from big data thinking. .
The principle of focusing on correlation: change from causality to focusing on correlation
Focusing on correlation instead of causality, society needs to give up its desire for causality and only need to focus on correlation, which means that it only needs to know what it is, not why. This overturns the conventions since ancient times, and our most basic way of making decisions and understanding reality will also be challenged.
In this uncertain era, when we find the exact cause and effect relationship and then do things, this matter is no longer worth doing. Therefore, the thinking in the "big data" era is a bit like the mechanical thinking that has returned to the industrial society-mechanical thinking means that when you press that button, the corresponding result will definitely appear. This is the state. As agricultural society moves forward, it does not need to find a very close and clear causal relationship, but only need to find the relevant relationship, only need to find the signs. The society therefore gave up the traditional preference of looking for causal relationships and began to explore the benefits of related relationships.
Buildings that are illegally partitioned inside the house are much more likely to catch fire than other buildings. New York City receives 25,000 complaints about overcrowded housing every year, but there are only 200 inspectors in the city who handle complaints. A team of analysts in the Mayor’s Office feels that big data can help solve this gap in demand and resources. The team established a database of all 900,000 buildings in the city, and added data collected by 19 departments in the city: tax arrears seizure records, abnormal use of water and electricity, payment arrears, service cutoffs, ambulance use, Local crime rates, rat complaints, and so on.
Next, they compared this database with building fire records ranked by severity in the past 5 years, hoping to find correlation. Sure enough, the type of building and the year of construction are factors related to the fire. However, an unexpected result is that there is a correlation between buildings that have obtained external brick wall construction permits and a lower incidence of serious fires. Using all this data, the team established a system that can help them determine which housing congestion complaints require urgent handling. The various characteristic data of the buildings recorded by them are not the cause of the fire, but these data are related to the increase or decrease of fire hazards. This knowledge has proved to be extremely valuable: in the past, only 13% of house vacancies were issued by house inspectors when they appeared on the scene. After the adoption of the new method, this proportion has risen to 70%-efficiency has been greatly improved.
Business people all over the world are chanting the advantages of the advent of the big data era: how a supermarket discovered the fact that a 17-year-old girl was pregnant from the shopping list of a 17-year-old girl; or put beer and diapers together for sale, it’s amazing Increased the sales of both parties. The information revealed by big data does sometimes subvert. For example, a statistics from Tencent on social networks shows that more than twice as many men love to watch family dramas than women; Chinese aunts care about the price of gold the most, but the post-90s are closely followed. In the past year, all of the top ten wireless payment ratios in Alipay were in Qinghai, Tibet and Inner Mongolia.
Forecasting principle: from unpredictable to predictable
The core of big data is prediction, and big data can predict in many ways. Big data is not about teaching machines to think like humans, on the contrary, it is about applying mathematical algorithms to massive amounts of data to predict the possibility of things happening. Precisely because in the face of the laws of big data, everyone's behavior is the same as everyone else's, with no essential changes, so businesses will change consumer behaviors more than consumers.
In addition, as the system receives more and more data, the system can be improved by recording the best predictions and patterns found. It is often seen as part of artificial intelligence, or more precisely, as a type of machine learning. The real revolution is not in the machines that analyze data, but in the data itself and how we use it. Once statistics and the current large-scale data are integrated, it will subvert a lot of our original thinking. So now there are more and more things that can become data, and the ability to calculate and process data is getting stronger, so everyone suddenly found this thing very interesting. So, what can big data do? Can do a lot of interesting things.
The Internet, mobile Internet and cloud computers ensure the possibility of big data real-time forecasting, and also provide enterprises and users with real-time forecasting information, and related forecasting information, allowing enterprises and users to seize the opportunity. Because of the full sample nature of big data, people and people are the same, so the efficiency and accuracy of cloud computer software forecasts are greatly improved. There will be this kind of result if there are signs of this kind.
The principle of information searching for people: from people searching for information to information searching for people
The development of the Internet and big data is a process from people looking for information to information looking for people. First, people looked for information, people looked for people, and information looked for information. Now is the era of information looking for people. The era of information seeking people means that, on the one hand, we have returned to a kind of original. The broadcasting mode is information seeking people. We listen to the radio and we watch TV. It is information pushed to us, but there is a flaw. I don’t know what we are. Who, later, the Internet did the opposite, providing search engine technology to let me know how to find the information I needed, so search engine is a very critical technology.
Big data has also changed information superiority. According to evidence-based medicine, the first thing to treat a disease now is not to study pathology, but to use past data to study how to treat under the same circumstances. This leads to the loss of the information advantage between experts and ordinary people. I used to believe in doctors because they know a lot, but now I can check on Google to find out what disease I have.
Google has a machine translation team. At the beginning, the translated text could not be understood at all, but now 60% of the content can be read. There is a joke in the Google machine translation team, saying that every time a linguist leaves the team, the quality of translation will improve. The more the expert, the more confused it is, but breaking the convention and letting the data speak, the faster the speed of obtaining the truth.
The principle that machines understand people: From people understand machines to machines understand people better
Not to make people understand the machine better, but to make the machine understand people better, or to be able to use the machine even when the user is stupid. It’s not even for people to understand the environment, but for our environment to understand us, and the environment to adapt to people. To a certain extent, the natural environment cannot say that, but in the digital environment, there is already such a trend, that is, the world in which we live. , It tends to be more suitable for us and understand us better. Which company can truly make machines understand people better, make the environment understand people better, and let the whole world of life we carry with us understand us better, then it must be competitive, and "big data" technology can help We can help.
One of the core goals of big data technology is to dig out the hidden laws from the huge and varied structure of the data, so as to maximize the value of the data. Computers replace people to mine information and acquire knowledge. The ability to quickly obtain valuable information from a variety of data (including structured, semi-structured, and unstructured data) is big data technology. In big data machine analysis, techniques such as semi-supervised learning, ensemble learning, and probabilistic models are particularly important.
Principles of e-commerce intelligence: Big data has changed the e-commerce model and made e-commerce smarter
Business intelligence has been redefined in today's era of big data. For example: After traditional enterprises enter the Internet, after mastering the application of "big data" technology, they will find a sense of sudden enlightenment. It is like looking for something in a dark room and can't find it. Suddenly they encounter a switch and find that it is so laborious. It turned out to be easy to find things. Big data thinking, in fact, is not a full name judgment, but a description of a certain latitude of our era.
The era of big data does not mean that our era has nothing but big data. Even in the Internet and IT fields, it is not everything. It just means that we have added such a clear light to the characteristics of our era, which leads us to think about the past. The state of existence, and a differentiated expression of our personal life state.
Of course, the same technology can also be used to diagnose diseases, recommend treatment measures, and even identify potential criminals. In other words, if you don’t know it yet, the medical examination company or the hospital reminds you to go for the examination as soon as possible. You may get some diseases. The business knows yourself better than you, and people like you will appear under certain circumstances. The possible changes. Just as the Internet has changed the world by adding communication functions to computers, big data will also change the most important aspects of our lives because it creates unprecedented quantifiable dimensions for our lives.
The principle of customized products: from the production of products by enterprises to customized products by customers
The next wave of reforms is mass customization, customizing products and services for a large number of customers, with low cost and personalization. For example, a consumer hopes that the car he buys is red and green, and the manufacturer is able to meet the requirements, but the price is not as unaffordable as handmade. Therefore, under the premise that manufacturers can afford the high costs brought by mass customization, to truly achieve personalized products and services, they must have a good understanding of customer needs, and behind this, they need to rely on big data technology.
In the era of Internet big data, merchants may finally be able to make precise price discrimination against each customer. Many of our current behaviors are relatively extensive. Airlines will give us mileage cards to accumulate mileage based on the number of kilometers flown, but in fact, different mileages flown by different customers have different contributions to the airline's profit. So one day a customer may receive a letter, "Congratulations, sir, you have been selected as a lucky customer, we have upgraded you to a platinum card in advance." This shows that this customer has contributed enough to the airline. . One day the bank said "Congratulations, your limit has been increased again," which means that too much money has been spent.
Precisely because of the laws of big data, everyone behaves like everyone else, without essential changes. Therefore, the business will be more of the consumer's behavior than the consumer. Maybe you are thinking, after a year of hard work, where do you want to go on holiday? Open e-Mail, and there will be mails from airlines and travel agencies.
Enterprise products are sold directly to users, eliminating the need for intermediary circulation, so that the price of the product can be sold at the ex-factory price, and if the sales fee wants to systematically learn big data , you can join the big data technology learning exchange deduction : 522189307 , welcome Add, understand that course introducers have gained benefits, online products have become users’ belief that cheap, and an online shopping market has been formed. In order for users to become fans of your products, you must understand their needs. Customized products become the wishes of users and become a new direction for enterprise development.