Remote Labor Index: Can remote projects be fully automated?

Using AI to improve workflow and efficiency as a programmer has become a necessity to compete in today’s market. But just how good are LLMs on their own?

This is what the Remote Labor Index aims to measure (link to the study and paper).

Remote Labor Index (RLI), a broadly multi-sector benchmark comprising real-world, economically valuable remote-work projects designed to evaluate end-to-end agent performance in practical settings.

And it actually looks like a very high-quality study. They claim to have compared projects that cost over $140,000 and took more than 6,000 hours of real work with their AI-generated counterparts.

Across a wide selection of different areas, AI attempted to achieve at least the same level of quality as a freelancer.

Result

So how good is AI? How many projects can be fully automated?

A measly 3.75%. You probably expected the number to be much higher, but that is the current state of AI right now.

Impact on programmers

Only 3.75% of projects can be fully automated right now. So programmers are still safe?

Well, I think yes. But keep in mind that different skills were tested. Maybe the AI performed particularly poorly at 3D animation, while programming was closer to 10%?

And that number is obviously only going to grow.

Conclusion

It is good to see such a low number. But it will continue to grow, so we as software engineers need to be aware of that and keep sharpening our skills.

As long as that number is not 100%, some work will still need to be done and our jobs are safe. Also keep in mind that software projects will become more advanced, more complex, and more time intensive.

So even if AI becomes responsible for coding 50% of a project, the other half might still require the same amount of effort that a full project requires today.