Why Semantic Layers? The Very Human Purpose of a Key Technical Architecture

This post was written by Donald Farmer. Donald Farmer is a seasoned data and analytics strategist, with over 30 years of experience designing data and analytics products. He has led teams at Microsoft and Qlik Technologies and is now the Principal of TreeHive Strategy, where he advises software vendors, enterprises, and investors on data and advanced analytics strategy. In addition to his work at TreeHive, Donald serves as VP of Innovation and Research at Nobody Studios, a crowd-infused venture studio. Follow him on LinkedIn.

Today, every business is a data business. With just a little application of technology, we become rich in data about transactions, customers, suppliers, partners, products, competitors, and the economy. And we have the tools to work with this data: excellent visualizations, powerful dashboards, useful reports are only a click away with modern applications. The goal – long held by analysts – of giving every business user access to intelligent insight seems irresistibly within grasp.

And yet … it’s just not happening. Modern Universal Semantic Layer technology may help. In this post, I will explain why I think this approach is more promising than earlier efforts.

The Elusive Goal

For years, the key challenge in Business Intelligence was simple to formulate but difficult to achieve: building the ‘single version of the truth.’ This means creating a centralized, authoritative source of business data. It’s structured for high-performance analytics and designed to eliminate discrepancies, ensuring all stakeholders have reliable, accurate, and current information.

Without this warehouse of consistent data, every department might produce their own erratic insights, resulting in error and confusion. But the architectural demands of the data warehouse were intimidating, especially in the days of expensive data storage, lengthy load times and cumbersome analytics.

Even when we come close to building these authoritative systems there are frustrations. The design process is demanding and slow to respond to users’ needs. And users’ needs seemed to change like the wind. For some time the answer seemed to lie in self-service applications: let the users meet their own needs! With well-designed modern BI platforms, consuming centralized data would be simplified and the design of new analytics could be left to the business. But then we were back with the issue of inconsistency.

In large part, we have failed because we tried to solve a human challenge as if it were a technology issue. Like the Horse Whisperer (who didn’t help people with horse problems, but rather helped horses with people problems) we need to look at this from another point of view.

Innovation is Semantically Messy

There are some very good reasons to want an orderly, well-managed, consistent version of the truth. If you sell products throughout the Americas, for example, you may have to work in over thirty countries, over 20 currencies and within numerous national and local tax regimes. For good management and compliance, you simply must build reliable geography, currency and tax dimensions. Numerous other aspects of your business, from employee records to consumer standards, will demand similar regularity.

However, in such a diverse environment, you will also want to innovate with new products, new marketing campaigns and new processes. Your social media teams and designers are unlikely to see the world ordered by the same categories as your accountants and HR. The more teams want to innovate, the more they will push beyond the analytics that your existing well-structured data enables.

Innovators do not just produce new products and services: they also think about customers and users in new ways. They redefine the semantics of their industry – what it *means* to be a customer, what *kind *of product this is. Our most innovative companies – think of Apple, Uber, Netflix, Amazon, Tesla – do so dramatically. Here is the clue to our new approach.

The Semantic Layer simplifies data concepts and technicalities for business users. They don’t need to alter underlying business data to work in new ways. For instance, envisioning a new demographic category, from Tweens to Silver Surfers, doesn’t require rebuilding underlying customer data. This can be quickly and effectively done within the Semantic Layer, bypassing the more formal demographics defined in the data warehouse.

To innovate is to re-imagine the semantics of your business and the Semantic Layer is where you can do that best.

Semantic Layers Unlock the Best of Both Worlds

Semantic Layers provide an elegant solution to reconcile the organizational demand for consistency and the need for creativity that pull organizations in opposing directions. As an abstraction layer sitting above raw data sources, Semantic Layers enable users to build the reliable structures required for data governance while enabling flexibility for innovation.

By curating business logic into reusable definitions, metrics, and data models, Semantic Layers establish a common language around information assets which ensures broad alignment and accountability around goals, progress and the current state of the business.

Some key capabilities provided by a Semantic Layer include:

• Business-friendly naming and definitions of metrics, facts, attributes, hierarchies and other data elements, using terminologies familiar to business roles (ex: Customer, Revenue, Region).
• Business rules and logic such as calculations, filters, categorizations and assumptions applied to the raw data to derive meaningful views
• Relationships and linkages between critical business entities such as customer, sales, and products.
• Aggregation and consolidation of distributed data sources into unified views and metrics.

The Semantic Layer connects raw data in its native environment directly to the business intelligence tools, analytics applications and other data consumption channels where users make decisions. In this way, the Semantic Layer acts as the gateway for organizations to extract value from their data assets by making data accessible, understandable and actionable.

Performance with Purpose: The Final Piece of the Puzzle

While Semantic Layers provide invaluable flexibility and accessibility to business users, that agility means little without the performance to match. With AtScale, the clue is in the name – a Semantic Layer architecture designed for cloud scale, optimized to handle large data volumes across dispersed sources for large numbers of users.

This is achieved through intelligent caching strategies, automated data engineering providing query optimization, and the ability to leverage the computational power of underlying data sources. By doing so, AtScale ensures that the agility they offer for innovation is backed by the robustness and speed that modern businesses require. The past limitations requiring stark tradeoffs between governance and innovation no longer apply.

Today data is not just an asset — it’s the language of business — so it’s time to consider the Semantic Layer as a tool for both governance *and* innovation. It stands not just as a solution to the challenges of today but as a foundation for the even more challenging opportunities of tomorrow.