← AI Feed
AI Feed

Fifteen departments had fifteen dashboards

Two firms measured what agreeing one meaning across their data is worth. One counted it in days saved on a blocked order, the other in the cost of a single task.

How we Organise

The cost of one task fell by almost ninety per cent

LinkedIn runs more than 300,000 Spark jobs a day. It had built its own push-based shuffle service into Apache Spark 3.2. It has now replaced that service with Apache Celeborn and published the comparison against what it built. Daily shuffle fetch failures went from about 750 to about 20. Fleet-wide, the shuffle cost of one task has dropped by almost 90 per cent, with compute cost down about 10 per cent and application runtime down about 14 per cent.

We tell a client that a platform which cannot price one unit of its own work has no way to show anybody a saving.

Consumption per unit of output is falling, or the reason it is not is recorded.

This is not the first source to bear on that position: the earlier one priced a solved coding task on a benchmark, where this is a production fleet measured against the implementation it replaced. The figures are LinkedIn’s own and carry no independent audit. Our test counts token spend against recorded outcomes. LinkedIn counts compute cost against Spark tasks. The account reaches the principle without touching the test.

How we Build

A credit limit used to block an order for 22 days

Rafael Domene is global chief information officer at the manufacturing company Cosentino. He told CIO that fifteen departments took part in serving a delayed order and each kept its own dashboards. The firm kept learning late that an order would not arrive. Cosentino has spent three years building one semantic layer over its existing systems, with the process-mining company Celonis. It aims to finish this year. Where a customer exceeded a credit limit and an order was blocked, the average delay was 22 days. The maximum is now one day.

We ask that a firm keep its entities and the terms it has agreed in one place, so that what a word means does not depend on which department you ask.

The firm’s entities, their relationships and their agreed terms are held in one ontology, extended as each use case needs it, and the interfaces the firm publishes resolve their terms against it rather than against a local glossary, and every state change the use case makes resolves through the ontology, with the system that owns each piece of state declared.

An earlier source bore on this one too. A published reference implementation of an operational ontology argued that the vocabulary is the easy half. Domene’s figures are his own account to a journalist, with no instrument stated. The 100 million dollars he expects by the end of the year is a forecast rather than a result. After three years the financial side of the business is still outstanding, so a programme of this shape says little about extending an ontology one use case at a time.