In the quasi-"AI era", how to measure the work efficiency and productivity of programmers?

From data to large model application, on November 25th, Hangzhou Yuanchuang Conference was held to share development tips.

Nearly 20 technology, financial and pharmaceutical companies have implemented new R&D performance management methods and achieved encouraging initial results.

Product defects reported by customers are reduced by 20%-30%;

Employee experience score increased by 20%;

Customer satisfaction scores increased by 60 percentage points.

Large models and AIGC technology have given rise to a new paradigm in software development, and have also made the complexity of R&D management rise sharply. Although research claims that generative AI tools such as Copilot Do tools really help organizations improve efficiency? ”

In the quasi-"AI era", "AI + R&D efficiency" is likely to become a high ground for enterprises to build core competitiveness. How to scientifically, comprehensively and accurately measure the work efficiency and productivity of developers and R&D teams is an important proposition in R&D effectiveness management.

In a recent study, McKinsey expanded and supplemented the existing two sets of productivity indicator models to build an end-to-end view of developer work efficiency and productivity. The report states that the method is easily deployed through questionnaires or process data embedded in R&D management tools, without the need to introduce a large technology stack or tooling equipment.

McKinsey: Developer productivity and productivity view

Based on the DORA indicator and SPACE indicator, McKinsey expanded and supplemented 4 opportunity-focused metrics (Opportunity-focused metrics), and classified and aggregated all indicators according to the system level, team level and individual level, and finally obtained the final A developer productivity and productivity view of how to improve the way your product is delivered and the value of that improvement.

Source | "Yes, you can measure developer productivity"

01 DORA indicator

DORA indicators are proposed by Google's DevOps research and evaluation team after years of research and analysis. They are the closest quantitative management framework to standards in the technology field. They perform well in measuring R&D results.

DORA indicators involve two aspects: throughput and stability, and include four key indicators: deployment frequency, change lead time, service recovery time, and change failure rate. When the DORA indicator returns unsatisfactory results, it means the cause of the problem needs to be investigated, which often takes a long time.

Source | "2022 State of DevOps Report"

02 SPACE indicator

The SPACE indicator was proposed by GitHub and Microsoft Research to enhance the DORA indicator. SPACE is the abbreviation for Satisfaction, Performance, Activity, Communication and Efficiency; each dimension contains several different indicators applicable to the individual, team or system level .

Satisfaction and well-being Satisfaction and well-being
Performance
Activity activity
Communication and collaboration Communication and collaboration
Efficiency and flow Efficiency and flow

Taking into account a personal perspective (especially developer happiness), the SPACE metric is a good indication of whether an organization is optimized.

03 Opportunity-oriented indicators

McKinsey conducted a detailed observation of the R&D process from multiple perspectives and proposed four opportunity-oriented indicators: R&D internal/external cycle time, developer velocity index, contribution analysis, and talent capability score.

Source | "Yes, you can measure developer productivity"

1. Inner/outer loop time spent in research and development: Inner/outer loop time spent

The report states that in order to identify specific areas for improvement, the complete software development process can be viewed as two cycles. The inner R&D loop includes activities directly related to creating a product: coding, building, and unit testing; the outer loop includes other tasks that developers must complete to push code to production: integration, integration testing, release, and deployment.

Source | "Yes, you can measure developer productivity"

For developers, the inner loop is the process of building products and directly generating value, while the outer loop, although necessary, is full of complicated and trivial matters. Therefore, from a productivity and personal experience perspective, companies should improve outer-loop tools and automation as much as possible so that developers can invest more time in inner-loop activities . Among them, top technology companies aim to have developers spend up to 70% of their time on inner loop activities.

2. Developer Velocity Index: Developer Velocity Index

The Developer Velocity Index (DVI) study is a survey that measures enterprise technology, work practices, and organizational support.

DVI involves 46 driving factors in 3 major aspects and 13 competency areas, and is obtained by the weighted average of these 46 impact factors, which can be benchmarked with peers. This comparison can help identify specific areas of opportunity, such as backlog management, testing, or security and compliance.

Source | "Research Report on Developer Velocity (DVI)"

3. Contribution analysis: Contribution analysis

Assessing an individual's contribution to the team's backlog (pulling data from R&D management tools like LigaAI and normalizing the data using proprietary algorithms) can help reveal trends that hinder the optimization of team capabilities and give team leaders a sense of output. Clear expectations to improve performance.

In addition, it also helps managers identify opportunities for personal skill improvement or training and rethink role/task distribution within the team. For example, whether quality assurance testers have enough work to do.

4. Talent capability score: Talent capability score

The score is a summary of an individual's knowledge, skills, and abilities for a specific organization, based on an industry-standard competency map. Ideally, organizations should pursue a "diamond distribution," where the majority of developers are in the mid-range of abilities. This can help provide insights into coaching and upskilling opportunities, and in extreme cases, may require a rethink of talent strategies.

#Write at the end

Last week, OpenAI announced a series of key technology and product updates such as GPTs, Assistants API and GPT-4 Turbo model, making the AI circle boil again.

It is almost foreseeable that AIGC technology and applications based on large models will gradually be integrated into the daily work of developers and R&D teams, becoming part of the team's DNA. Facing the menacing wave of AI, R&D managers are urgently required to establish a scientific measurement index system to gain more intuitive insight into the work efficiency and productivity of developers and R&D teams. Perhaps this will answer more clearly:

What are the barriers that prevent programmers from doing their best work?
To what extent do culture and organizations influence a developer's ability to create great work?
How do you know if a programmer's time and energy are being spent on activities that truly drive value?
How do you know if your organization has all the development talent it needs?

LigaAI will continue to share content such as R&D efficiency management, practical R&D management practices, and advanced technology management growth. Welcome to follow us.

**Don’t forget to click on LigaAI - a new generation of intelligent R&D collaboration platform to try out our products~~~~~~**