Mengchen sent from Aufeisi Qubit
| Public Account QbitAI
The cost of GPT-4 to replace junior data analysts is only 0.71%, and it is 0.45% to replace senior data analysts...
You read that right, it is 0.71%, not 71%.
According to the Singapore market, a senior data analyst with an annual salary of 86,000-90,000 US dollars (600,000-630,000 RMB) will only need 300-400 US dollars (more than 2,000 RMB) to switch to GPT-4 .
This conclusion comes from a new paper by Alibaba Dharma Institute and Nanyang Technological University in Singapore , and is rated by netizens as a must-read paper for those interested in the field of AI and data analysis.
Specifically, middle and senior analysts in the conclusion refer to data analysts who have many years of work experience in the financial industry.
The performance of GPT-4 is comparable to a human being with 6 years of work experience in most indicators , with lower accuracy than human beings, but higher complexity and consistency indicators than human beings.
In comparison with another analyst with 5 years of work experience, GPT-4 lost to humans in terms of correctness of information, aesthetics of charts, and complexity of insights.
If compared with a junior analyst with 2 years of work experience, GPT-4 performs better in correctness and can complete more work.
But GPT-4 does all types of tasks much faster than humans.
On the assumption that there are 21 working days per month, 8 hours of working time per day, and wages are paid at market prices, the final conclusion is drawn.
What can GPT-4 do as a data analyst
The paper focuses on the following abilities of GPT-4 as a data analyst:
Generate SQL and Python code
Execute code to get data and charts
Analyze data and draw conclusions from data and external knowledge sources
Experiments with 200 samples show that for the task of drawing charts , GPT-4 can understand the meaning of instructions, and has a certain background knowledge of the chart type, so as to draw the correct chart.
Most of the charts are clearly visible without any formatting errors. The aesthetics index of the icons has a full score of 3, and the average score of GPT-4 is 2.73.
However, manual inspection can still find some minor errors. The graph accuracy index has a perfect score of 1, and the average score of GPT-4 is 0.78.
The paper specifically states that their evaluation criteria are very strict, as long as there is any error in any data or any label on the x-axis or y-axis, points will be deducted.
For data analysis tasks , GPT-4 averaged perfect scores for consistency and fluency, verifying that generating fluent and grammatically correct sentences is definitely not a problem for GPT-4.
Interestingly, the accuracy of the data analysis step is much higher than the accuracy of the chart information, indicating that although GPT-4 drew the wrong chart, it analyzed the correct conclusion.
In the case analysis, the research team also concluded three main differences between GPT-4 and human data analysts:
Human analysts can express their personal thoughts and emotions , such as writing "It is surprising that..." when analyzing; human readers can easily understand from such expressions whether the data is in line with expectations or abnormal.
Human analysts tend to draw conclusions in combination with background knowledge , such as writing “… is common in…”; GPT-4 usually only focuses on the extracted data itself, which can be improved by allowing GPT-4 to search the Internet for real-time online information.
When providing insight or advice, human analysts tend to be conservative , such as stating "if the data is OK..."; GPT-4 will directly give advice in a confident tone, without mentioning assumptions.
In addition, the team said that due to limited budget, it is mainly because it is too expensive to hire a senior analyst to compare with GPT-4, and the number of manual evaluation and data annotation is relatively small.
The final conclusion is:
Experimental results and analysis show that GPT-4 has comparable performance to humans in data analysis, but whether it can replace data analysts requires further research to draw conclusions.
Paper:
https://arxiv.org/abs/2305.15038
For more exciting content, please pay attention to the official account Programmer Xiaohui: