This post from my friend Aaron prompted (who is the LLM now?) me to take a beat on where I am personally with LLMs.

In a previous post I shared how I couldn’t tell if i’m more productive. I even made the case that productivity was an illusion, but things have changed. I have in the last three weeks checked in more code written primarily by Claude than in the previous 10 months combined.

Strata Business Intelligence

The Illusion of Productivity with LLMs

Every couple of months I attempt to prompt a subsystem for Strata into existence. This is different from having it do small tasks, api lookups, project planning, and other pointed code development (single methods or classes). It’s been a game changer for these tasks. A subsystem, on the other hand, is an entire featureset from the ground up. I’ve bee…

Read more

3 months ago · ajo

Claude Opus has been a game changer for us and things are moving faster than expected. The change is not entirely due to the model alone. We are getting really good at leveraging it appropriately. Our team was able to produce a stunning CLI for Semantic layer management in record time. And, I was able to finish boring parts of the app (ie System settings and monitoring UI) faster than my most aggressive estimates. Another explanation of the speed here could be that both creating CLI’s and standard CRUD interfaces is pretty simple. But that is exactly the point. We are able to off load common app parts easily to the LLM. More complex features need more care and management but this definitely accelerates our pace.

What took me 2 years to build at Netflix is nearly feature parity in under a year with even more capability.

Greenfield Projects and LLMs

We are building a completely new app in fresh repo’s. I’m acting as a PM, UI Designer, Frontend Engineer, Backend Engineer, and architect all at once. This means context muddling for me and the agents. The temptation is to one shot entire features and call it done. The problem here is that one shotted code is high entropy. I’ve thrown away tons of one shotted features and handwritten them from scratch. The better approach for me is to segment usage of LLM by the role I’m performing rather than the feature I’m building.

What does this mean in practice?

First, I take a feature I want and spec it out completely from the end User experience POV. This is the Product Management role. I’ll take this spec and one shot it. Here I blindly accept all the code and don’t bother reviewing it. The goal is to figure out if the product spec is right. Does the UX make sense? Do we need sub features to better describe the main objective here? Or, is the feature completely wrong for our target use case. This is a game changer! You can protype features before even considering them for the final product. Throw away work like this could only be done with large teams and big money in the past.

From here, I’ll either iterate to test different experiences or start working on the architecture. To do that I’ll review how the LLM thought about it then re-architect it from first principles as it relates to the long term vision of our product. The routing paths, main files, code structure, and models will be outlined by me. Then I walk the LLM through implementing each in groups that make sense. For example, for precaching data lists, we have a model called Elemental and background job called ElementalJob. That whole implementation can be done together. Once the backend is done we move to the frontend and follow the same process.

This approach is easier for me and the agents to understand. Its step by step but 10x faster than what I alone could have done in the past.

Some will argue that I’m wasting time and I should just accept the one shotted code. I’m sorry I just can’t. I really care about the aesthetics of the entire app. From the frontend to the backend. I care what the URL looks like, what the class names are, and how the code is organized. I want it all to be aesthetically pleasing. And guess what? I can have it all now with the help of LLMs. This sort of thing would make me slower in the past but I can get it for free now.

Lastly, being focused on end to end aesthetics will pay dividends in the future. My code will be easier for agents and humans to understand and it will become faster and faster to add capability.

Subscribe now

Every couple of months I attempt to prompt a subsystem for Strata into existence. This is different from having it do small tasks, api lookups, project planning, and other pointed code development (single methods or classes). It’s been a game changer for these tasks. A subsystem, on the other hand, is an entire featureset from the ground up. I’ve been running this experiment since April 2025. Now, a full eight months later I was finally able to merge code from Claude for a somewhat substantial module. With that revelation, over the next two days I attempted a slightly more complex module. And like every previous attempt this was a fail. All of that work discarded. Instant regret! For startups, time is critical and wasting two days of work feels awful.

This has been an ongoing challenge for me. Two steps forward and two or more steps backward. I was able to build the same complex module with more feature completeness, less lines of code, and better performance in less than a day.

Maybe I’m wrong. I’m having a perception issue. I could be subconsciously biased against AI’s coming to take our jobs or whatever. You be the judge.

Background

We are building a semantically enabled Business Intelligence application. Our primary focus is on enabling non-technical domain experts (think business analysts, operations folks, finance analysts etc) to maximally self serve data. I first built it at Netflix where it was unbelievably successful for this use case. In fact, over my decades in the industry that was my third attempt at solving this specific problem.

At Strata, we are taking the learnings from that experience and building an even more amazing BI platform. Our tech stack is Ruby on Rails for both frontend and backend.

This being the fourth time I’m building such an application and having recently built it at Netflix, means I have a significant advantage. In fact, in terms of building semantic engines I might be one of a handful of experts in the world.

Now you know what we are building, my advantage, and our tech stack. Let’s take a look at how I’m using AI.

Day to Day with LLMs

Everyday while working on Strata, I’m using both Grok and Claude. I find Claude more capable for coding tasks in Ruby. Grok on the other hand is better at doing deep research on specific topics. I’m going back and forth between these two. But in general, 80% of the time I choose code output from Claude over Grok. On the backend I use Claude Code and my code editor and setup is NVIM.

These are all the successful ways I’m using LLMs:

1. API lookup / Search. I rarely have to double check the original documentation. Although every now and then it still does make up things that doesn’t exist. This basically replaces search. Rating: 9/10

2. Sample Code Gen. For example, sample code for manipulating a complex multidimensional array in a specific way. I then take the code and modify it for my needs. In the past, I might be browsing stack overflow for examples. So far, only occasionally does it surprise me with a better implementation than I would have come up with. I still have to modify it quite a bit for my specific needs. Rating: 7/10

3. Translation. Translating an existing implementation for other use cases. For example, I have a YAML config system that defines Strata behavior for each target database type. LLM’s could easily translate these YAML for all our target databases with pretty high accuracy. Only issue is you need to double check because there are subtle mistakes. Rating: 7.5/10

4. Refactoring. Assisting with refactoring is huge. It works really great when you need to rename classes or relocate them. It’s pretty good at finding all the references even when variable names are not substantially related. Its basically a supercharged search and replace. Rating: 8/10

5. Code Review. For somewhat complex classes and methods I’ll have it review my work for obvious improvements. Its somewhat mediocre here. But a good sounding board. Rating 5/10

On a day to day usage for these small tasks it looks like I’m getting a huge productivity boost. How about for a moderate subsystem?

Formatting Module

Over Christmas break, I spend about a day and half building a new formatting system with Claude. In Strata, data can be formatted based on data types, rules at the semantic layer, or rules at the reporting layer. We need to support number and date formatting in addition to advanced formatting via HTML templates and Javascript functions.

Additionally, the formatting module has to apply the formatting rules for backend processing, export formatting rules for client side formatting, export excel and google sheets compatible formatting, and parse formatting rules into appropriate JSON structures.

This is not a super complicated project but it is substantially more than writing a single code snippet. I wish I had tracked the exact amount of time I spent on this. This was over the holiday break and I was squeezing in work while kids were sleeping or away playing. I’m estimating about a day and half to maybe two days.

It required two complete builds before getting it close to correct. Iteration one was way over the top. It didn’t use any of the out the box formatting features that Rails provides. Claude built everything from scratch for some reason. I try to prompt it back to simpler solution but the code just got messier.

Having learned how Claude responded to the first prompt, I created a second prompt (these are detailed markup docs not one liners) with specific instructions on simplicity and usage of Rails existing capability. This worked pretty well. I probably refactored about 20% of the code for correctness, clarity, and organization. This is the largest chunk of code that I have commited from AI so far. And this is the only module/feature level code I’ve accepted from an LLM.

Am I truly satisfied with this implementation? I’d say no. I probably have at least one more refactoring to do for correctness. This is mostly around the use of locales. And, to get the right locale other parts of the system needs to modified. Claude never considered locale at all.

Feels like success on the surface. But, I can see the tech debt. Debt that will come due soon.

Are my feelings here a perception problem? I feel like I could’ve built a more complete system in the same time or with an extra day or two. Maybe I’m over estimating my capabilities. Let’s explore my next LLM project that is substantially more complicated that I ended up rewriting. This may answer the questions posed here.

Share

Chart Rendering Module

Strata is a Business Intelligence tool. Of course, we have to take raw datasets and transform them into beautiful visualizations. Our target users just need basic chart types: line, bar, area, donut, stacked area, and stacked bars. Maybe a few more. We already have systems to generate queries, execute against a data warehouse, cache results, and reuse cached results. We also have a beautiful editor where users can configure how they want the chart to render (x, y, and series axis. Tooltips and row axis).

Now we just need to transform the data to be rendered by Chart.js. This transformation requires understanding how to leverage series when its a dimension vs when its a measure. It has to understand how to handle multiple measures on the y axis and what to do when dual axis is enabled. Finally, it needs to create chart multiples when row axis is configured. Much of the work is transforming data into Chart.js compatible JSON object. As a bonus, we also need to enable color themes for rendering many series.

I made three attempts at building this from scratch. Each time refining the detailed spec describing everything from code style to requirements. On the third attempt I’d estimate we reached 70% feature completeness, but only 50% correctness. Complete means it responds to a certain scenario with an output that looks right. Correctness is how robust the the feature is against bugs. This is my qualitative assessments so you can take it with a grain of salt. Neither correctness nor completeness are the reason this attempt failed. In fact, the chart rendered beautifully with color themes and formatting. With some more prompting and refining I was able to improve correctness and completeness. However, I did note the lines of code and structure seemed more complex than what I would do. But, I let it proceed as I was intent on getting to success here. Only to end up scrapping the whole thing.

So what happened?

Well in certain situations the chart was taking over one second to render even with fewer than a 100 data points. Most of the second day was spent prompting our way into diagnosing the issue. The one second to render was pure client side times. Nothing to do with the server response (although that wasn’t amazing). In the end, neither I nor Claude could figure out why this was happening. Hours of prompting and no luck. I know the Chart.js library well and in no way should it take 1+secs to render.

Instead of deleting all of Claude’s work as I normally do in this situation, I decided to create a fresh branch of master and hand roll this whole module. Good for comparison. In one day I was able to get to 50% feature completeness and 90% correctness. (Correctness is subjective, there maybe bugs I have not found or thought to look for). I probably wrote less than 20% lines of code compared to Claude and only needed two new files. Whereas, Claude had 6 new files and way more lines of code. But the biggest win, only 14ms to render 1000+ data points on Chart.js. This is without doing any special performance optimization. I still have no idea why the Claude implementation had a problem here.

I lost two days of productivity here. But was I faster on my hand roll because of those two days? That’s what makes it hard to tell. AI’s are becoming part of our habit which is making it hard to tease out the gains.

Summary

My ongoing experiments with full module implementations are erasing some of the gains I’m making with day to day use cases listed above. If I stick to those simple tasks, I would say I’m 3x more productive. But letting AI take on bigger tasks is setting me back days. I don’t have an intuitive feeling of being more productive hence why I’m leaning toward neutral to negative productivity gain.

Maybe I should stop these experiments. But I can’t, nor can you. There is too much influential hype about what others are able to achieve. This leads to a nagging feeling that your not prompting right. That its your fault. Or, that the new version is substantially better and can finally get the job done.

Or, it’s all an illusion. We have a talking machine that tells you things with great confidence. That kind of confidence plus a sense of authority is persuasive.

Share Strata Business Intelligence

Personas, personas, personas. Unlike consumer software, Enterprise Applications target users with diverging skill sets. The same app could be used by expert developers and non-technical business users. These applications are not always like Google Sheets where everyone is using the same functionality. Instead, they often have features that can only be used by experts. This is especially true when it comes to Business Intelligence (BI) tools.

The simplest BI tool may involve the following set of users:

1. Administrator [Moderate Technical]

   1. More technical than an excel ninja but not necessarily a developer/programmer

   2. Responsible for set up, maintenance, and user management

2. Data Engineer/BI Engineer/Analytics Engineer [Technical]

   1. Highly technical user responsible for configuring data artifacts

      1. Connections

      2. Semantic models (if applicable)

      3. In memory data caches and related pipelines

   2. Develops key dashboards and reports

3. Business User [Non-Technical]

   1. In many cases these users passively consume dashboards created by other personas

   2. Some subset of these users will try to self serve basic dashboards and reports

Take a close look at (1) and (2) above and compare that with (3)’s you know. The skills set gap is continental. Now imagine building a tool while trying to optimize for all three personas. Then imagine adding data scientists and other personas as vendors seek to increase market share. Many spinning plates stacked precariously on a single tool.

This is why your company has 10 BI tools and you hate them all.

Read more

The original idea behind Business Intelligence (BI) tools was to enable self service reporting for non technical users. Over the years BI tools have picked up many more features and feel less and less suited for business users. We went from drag and drop report building to highly complex dashboards that can only be created by platform experts. Vendors sell BI tools with the promise of “self service”, but companies quickly realize many of their requirements can only be fulfilled by experts. We are not talking just about common technical operations like semantic modeling, server deployments, data source configuration etc. Expertise here makes sense. Instead, the core self service utility itself has become over complicated and require advanced skills.

To illustrate, imagine I’m a business analyst in the Customer Service org and I’m coming into an existing Tableau server environment. I just want to know a little more about a KPI. Where do I even start? There are hundreds (often thousands) off data sources and enumerable workbooks. I can’t even start without a guide or manual. Sure sure, you have to do some work, everything is not going to be handed to you on silver platter. But simple things should be simple! This is the kind of inertia you have to get over to make your organization truly data driven. (I don’t mean to pick on Tableau, it’s just popular and well understood. They all have similar issues.)

Vendors often promise enablement for business users, but in reality they are reduced to mere passive consumers of static data. Here experts create reports and dashboards for the masses. This is fine for a class of use cases but it is horrible for your operators. These are the users running your business, purportedly making data driven decisions day in day out. For them this is anything but self service. Basic questions that drill down even to shallow levels often require an engineer in the loop. No question that this is slowing down decision making.

How did we get here?

Feature bloat over the years have largely been thoughtless and reactive. This is the result of a highly competitive and fractured market. You could easily list the top 25 with ChatGPT and you would not have heard of many of the names.

Here is a list of less well known BI tools from Grok:

1. Cognos

2. Sigma Computing

3. Hex

4. OBIEE

5. Evidence.dev

6. Lightdash

7. Metabase

8. Pentaho

9. Pyramid Analytics

10. FineBI

Some of these are startups and some are legacy tools that still hold significant market share. How many of these have you heard of?

All of the tools start with an amazing core that solves a particular problem in a novel and useful way. But to grow their market share they quickly and thoughtlessly add competitor features. Over my 25 years in the industry I’ve seen many tools come and go and watched vendors adapt. As soon as a competitor shows up with a threatening proposition it is copied. Often, poorly just to check a box during a sales cycle.

Maybe we (buyers) are the problem

Customers go into buying mode with so many requirements that many are at cross purposes with each other. We want data prep, data viz, data exploration, semantic models, etc etc. But, we don’t spend time to understand who will benefit from each of these features. And we certainly don’t think about how these features should be delivered. For example, are the users of data prep capabilities and semantic reporting the same? If not, how will each user interface with a tool that has both in a coherent way. Long story short, they are totally different and vendor implementations of each will be too shallow to satisfy anyone.

Another bigger problem from the buying side is the team/persons with outsized influence on the final decision. Are they the right influencer for the target problem?

Read my post on how Lookers rise is mostly driven by data engineering rather than the end users:

Finally, many larger orgs with two or more existing Business Intelligence tools go into the market overly focused on consolidation. When this is the overriding impetus, you end up buying tools with all the bells and whistles on paper. Only later do you find out that these belles and whistles are too immature to replace the existing tools. Now you have 3 BI tools. Congratulations, welcome to the fortune 500.

Or, Maybe its the Market Research firms

BI tools, according to Gartner must have the following functionality:

1. Automated Insights

2. Data Preparation

3. Data Visualization

4. Manageability

5. Product Useability

The last two are no brainers for every tool. Not even sure why its on a list of must haves. Anyways, each of the functionality above is doing a lot of work. When you dig in, they are super broad and range in capabilities and skills match. Just looking at the top three it’s obviously at least 3 different types of personas. Maybe more when you get into the details.

What skills will the data prep persona bring to the table as compared to data visualization? Automated insights sounds like something fit for data scientists with specific expertise.

It’s probably fine if a platform offered all of these features. The problem is how it’s delivered and for whom. If there were sharp lines between the modules in terms of functionality and user match, then it might work. For example, when you get Google Workspace it comes with Docs and Sheets. But, they are two totally separate modules that serve specific use cases. Imagine if Sheets was a feature of Google Docs. This would be a horrible product. Instead these products are only loosely integrated and not dependent on each other. BI platforms tend to tightly couple all of their functionality. They feel bloated and not targeted for any specific persona.

A Time to Unbundle

You are going to buy more than one BI type tool for your organization anyway. You might as well get the best for each problem you are trying to solve. Imagine a tool that is only focused on delivering the best interactive dashboarding capability. No data prep, no automated insights, no semantic reporting. Just the best dashboarding tool for Analytics Engineers. This will be completely different from the best dashboarding tool for non technical business users. Combining these two personas is how you end up with tools that nobody likes.

The other benefit of focusing on specific tools for specific problems is that you don’t have to pay bloated prices for bloated software. Perhaps your total cost might remain the same but your productivity in each problem space will be exponentially improved.

It’s time to consider unbundling your BI stack. What do you think, should vendors start unbundling core functionality?

Leave a comment

Looker started in 2012 but already by 2014, only 2 years later, it was gaining significant market share and mindshare as the go to self service analytics tool. And in 2019, 5 years later, it was acquired by Google for $2.6 billion dollars. An amazing story by any account. Looker’s success was fueled by its LookML tech which allowed engineers to configure a semantic layer. It repopularized the concept of a semantic layer which at the time was losing mindshare.

Not sure what a semantic layer is? Read:

But, when you dig deeper its semantic layer left much to be desired. Barely useful.

1. Each explorer defined a single fact domain.

2. No concept of snapshot measures

3. No ability to create leveled measures

   1. Exclude certain dimensions from the group by or filter or both

   2. Include certain dimensions in the group by

4. No automatic way to blend data across fact tables

5. Custom syntax with a long learning curve to define semantic objects (LookML)

Looker’s semantic layer is good at querying a single fact domain at a time. But is that really a hard problem? Does that really enable self service?

At the time there were many tools that could handle much more complex use cases. Even more that could handle the commonly required features listed above.

So what happened?

A Case of the Tail Wagging the Dog

Maybe it was the amazing UX for business users.

No. Obviously, not that.

In fact, enabling business users seem to the secondary use case not the primary. A further surprising fact is that companies were paying a premium for Looker. But why?

Well friends, when the tech team, in this case Data Engineering, is more influential in the buying decision, you end up with Looker.

Kudos to the Looker team on finding this angle though. Enterprise sales is extremely hard, but if you can find the right influencers for your product your chances to close goes up dramatically.

Looker solved many of the problems engineering teams were experiencing with traditional BI tools. These old guards of the self service analytics were exclusively GUI driven. Making standard principles of systems engineering difficult. Namely, version control and automation. This was enough for Looker to eventually end up with 2% of the BI market.

The Way Forward

At Strata we are solving for both. Smart adoption of software engineering principles for engineers and a world class semantic layer for business users.

We can have both!

The Ruby ecosystem has never been great with Data Engineering and Analytics. Python dominates here. Nothing wrong with that. I believe in best tool for the job. However, when we are building our Ruby apps, whether Rails or otherwise, we need to be able to connect to popular data warehouses for certain use cases.

Introducing dwh, a lightweight adapter providing a unified interface across popular data warehouses and databases for ruby. It is not an ORM and is designed be simple enough to quickly add new adapters.

Why didn’t we just contribute to Sequel?

Libraries like Sequel are amazing and comprehensive. However, its broad coverage also makes it more laborious to add new databases. We seem to be having an explosion of databases recently and a lightweight interface will allow us to integrate faster.

The adapter only has 5 core methods (6 including the connection method). A YAML settings file controls how it interacts with a particular db. It is relatively fast to add a new db. See the Druid implementation as an example. And here is corresponding YAML settings file for druid.

Core Methods

Six core methods rounds out the main functionality we need when deally with data warehouses. There is no direct attempt to type cast results into Ruby types. However, some of the adapters use existing gems like pg and mysql2, these provide type casted result sets. You can get back native results like so:

adapter.execute('select * from customers', format: :native) 

connection - Creates a reusable connection based on config hash passed in

tables(schema: nil, catalog: nil) - returns a list of tables from the default connection or from the specified schema and catalog

metadata(table_name, schema: nil, catalog: nil) - provides metadata about a table

stats(table_name, date_column: nil) - provides table row count and date range

execute(sql, format: :array, retries: 0) - runs a query and returns in given format

execute_stream(sql, io, stats: nil) - runs a query and streams it as csv into the given io

Tutorials and Guides

• Getting Started

• Adapter Configuration

• Creating an Adapter

• API

Quick Start

# Install gem
gem install dwh

# Create an adapter instance
require 'dwh'

postgres = DWH.create(:postgres, {
  host: 'localhost',
  database: 'mydb',
  username: 'user',
  password: 'password'
})

# Execute query and return results as row of hashes
postgres.execute('select * from customers', format: :object)

An Active Abstraction

Let’s get into the tech first. Semantic Layers bridge the gap between your business concepts and your physical database. For example, suppose you are running a customer service team with call centers etc. You might be tracking a common metric called First Contact Resolution Rate by Country. Here the business concepts are First Contact Resolution Rate (FCR) and Country. These then map to a database called cs_dwh, a column in the contact_f table called country_origin_iso_code, and a metric defined as sum(first_contact_resolution_cnt)/sum(num_contacts). (Don’t worry if this is confusing, clarity is on the way). Notice how much more transparent and simple the business terminology is versus how the data is actually stored.

You could simply document this mapping and call it a semantic layer, and some do, but that would only be useful for highly technical users.

What we have in Semantic Layers is an Active Abstraction. Its active in that, it can take user input like “Last 30 day FCR by Country”, connect to the appropriate database, construct and issue the correct query, and return results in meaningful formats.

If you didn’t know anything about Business Intelligence or Semantic Layers, you can understand the potential here. You start to wonder what if everyone in your company has fast access to high quality data? And, what if all our data needs are intermediated by a Semantic Layer so we can have a single source of truth? At first it looks like many of your problems with governance and self service could be one shotted with a Semantic Layer. But this is also where your problems start. Going down this road to extreme adoption of Semantic Layers will lead to frustration and eventual abandonment.

Your thinking “how does he know this”? Well dear reader, I’ve been working in this industry for over 20 years and I’m obsessed with Semantic Layers. This will be the third time I build a Semantic engine from scratch. I might be one of a handful of experts in the world who understand how to build one and how to deploy it successfully. Before we get into succeeding with this technology, let’s dive into why extreme adoption will lead to failure.

What if Everyone has Access to Data?

“Democratize data!”

“Every decision is now data driven.”

“You must use the enterprise standard Semantic Layer for all your analytics need.”

It always starts with so much promise and then you take it too far. Now millions of dollars in investment are wasted and the users are gone. Where did we go wrong?

First, you didn’t match the technology to the right users and domains. Instead, you built for everyone and every domain. Semantic Layers, when done right, enables a level of self service that is closer to raw data than curated dashboards. Also, besides the basic semantics, it is mostly context free. This means if you are not an operator within the domain in question, you could easily get lost.

Second, teams try to build a cross domain Semantic Layer across the entire enterprise. This goes wrong a number of ways. Everything from the politics of semantics (who gets to call their metric “Revenue”) to the variable quality across domains. Additionally, it turns out only a minimal amount of cross domain insights are useful in a self service context. The most enabled users are operators. They are primarily interested in the domain they operate in. They need to go deep not broad. Teams in this situation end up building broad rather than deep.

Third, you mandated the use of one tool to rule them all. It covers your exploratory visualization needs, your interactive dashboarding needs, your data science needs, and your self service reporting needs. All built on top of a universal semantic layer. Guess what, it sucks at every one of those use cases. Its optimized for no one’s happiness. A person who is capable of exploring the raw data via direct querying has little need for a Semantic Layer. Same goes for the Data Scientist and Engineers. For them, a Semantic Layer is a means to bootstrap queries rather than a direct interface to the data.

The Semantic Middleware

“Lets take all of our access layers (BI Tools, Visualizations, Notebooks etc) and make them query through the Semantic Layer”.

“Single source of truth!”

“No more governance issues.”

This is going to fail harder than federated data warehousing from a few decades ago. A couple of vendors and a small cadre of Tech companies are trying to make this happen. Long story short, it ain’t happening.

A Debugging Nightmare

When you stack abstractions upon abstractions — as is the case with BI tools connecting to a third party Semantic Layers — you better be prepared for debugging marathons.

The BI tool in question will of course have its own abstraction as it maps your data to its internal model. It is designed to work without such a middleware to begin with. Now when the CEO asks why Revenue on his pretty dashboard is wrong where do we look:

1. The metric as defined in the Dashboard

2. The metric as defined in the BI tool

3. The query generated by the BI tool

4. It could be how the metric was interpreted by the Semantic Layer. I.e. It knows of a multiple metrics called Revenue and chose the wrong one.

5. The metric as its defined in the Semantic Layer

6. The query generated by the Semantic Layer is wrong

7. The data in the actual database is wrong

Some will say only 7 places to look, not bad. Good luck to you!

It can’t be the Single Source of Truth

As mentioned above, not matter what, your BI tools need to map your data to its own internal model. Take Tableau for instance. Even though you connect to data via the Semantic Layer, you still have to define local metrics in the workbook. What is stopping anyone from changing metric definitions using components from the Semantic Layer? That’s right, nothing!

One of the main reason we have inconsistent metric definitions is people. Not tools, or documentation or some process. Its people and politics. And most of the time they have a good reason to have a flavor of metric that is slightly different. This should be totally fine! We just have to be good about using different names.

Share

Performance will Suck

If not today, eventually. It will become expedient to solve problems with your data model in the Semantic Layer. Issues that should be a proper ETL or Data Model change will come find their way into the Semantic Layer. Most current Semantic Layers are already slow (5 seconds or longer per query) by its nature. They are designed to query against data warehouses which in the context of big data are not meant for sub second response times. You just can’t deliver high performance interactive dashboards given the latency.

Well you could, if you wanted inflicting suffering on your colleagues.

To address this need to power interactive dashboards semantically. Vendors will pitch their proprietary in memory OLAP solutions (aka cubes). As of now these tend be incoherent with the existing Semantic Layer. Meaning, you can derive cubes from the base layer but they operate independently. You have to direct queries at the cube and or the cube has its on lite semantic model which differs from the base model.

Even if you could solve the performance issue with cubes, it doesn’t matter in the end. A capable Analytics Engineer building high impact interactive dashboards will only be slowed down and held back by the Semantic Layer. I’m talking about executive dashboards here with broad visibility. Not one off interactive set of charts.

You’ll pay for performance one way or the other. Slow dashboards or long development cycles or both.

So.. Semantic Layer Bad?

Absolutely not!

The Semantic Layer is one of the highest leverage investments you can make on your data assets. It is the penultimate debugging tool for your business. Today, most non-technical users access data via interactive and static dashboards. However, less than 1% of data is accessible via these modalities. You didn’t spend millions of dollars per year to move, process, and store data just for it to sit idle. Lets unpack the real value here.

Subscribe now

The Right User: Non-Technical Domain Experts

At Netflix, where I successfully built and deployed a Semantic reporting tool, I coined the term “Non-technical Domain Experts.” They are an underserved class of users who are hungry for data. Interactive dashboards are less than useful for them. They are operators who directly influence business decisions in real time. Some are charting a new course for your product. And others are tracking down that last dollar.

Non-Technical Domain Experts

While they may not be technical in the sense that they can’t write an ETL pipeline or even basic SQL queries, they are masters of their business area. Whether its a subdivision of a subdivision, a whole department, or a product line, these folks are all over your company. When shit hits the fan, these are the folks that tell you what’s happening on the ground from a data perspective.

They understand the KPI’s and associated metrics better than the technical team who put them together. Your data engineers understand the technical nuance of a metric, these people understand the business nuance.

You’ll find them influencing or making key business decisions. Decisions like where to deploy staff, how much inventory to buy right now, which bugs to fix tomorrow, and how much revenue will product A land this month. They might have “Analyst” in their title but that’s not always the case. And finally, they are often wizards with Excel.

Now imagine that this cohort in your company is not limited to simple dashboards. Nor, are they sitting around waiting for engineers to answer basic questions. That they can dig deep into your data to make the right call at the right time. This my friends is the real unlock.

The Right Use Cases

Can you build interactive dashboards on a powerful Semantic Layer? Sure, but you should not optimize for that. The class of users and use cases served by interactive dashboards are different. Dashboards are designed to summarize KPI’s and disseminate data to a broad user base. Much will be done to improve performance and usability. These exigencies will clutter and confuse your Semantic Model.

Generally, operational reporting is an obvious top use case. The only thing I’ll say here is that at the enterprise level you should think carefully before relying exclusively on canned reports that come with your solution. These are often devoid of rich data that is available in your data warehouse.

Here’s how to find other potential users:

1. They frequently go from KPI’s to transactional data

2. They frequently need custom filters and groupings added to existing dashboards and reports

3. They want to download the entire dataset backing a dashboard

4. They are constantly making requests for custom data pulls

These are some of the right targets and they are in almost every domain.

Determinism for your Probabilistic AI

We are in the midst of an epic tech cycle lead by Artificial Intelligence. Computers can finally talk to us and we can talk back in plain english. This is amazing and has done wonders for my productivity as it likely has for you.

However, as we all know by now there is a huge caveat. Large language models that power our current generation of AI tech are probabilistic by nature. That means answers to a question are non deterministic. Even if you had the funds to train your entire data warehouse, it will some of the time give you the wrong answer. In fact, it will give you different answer just for merely phrasing the question differently. And sometimes the answer to a question can be different between model upgrades.

Your CEO doesn’t want you to guess how much Revenue the company will close this month. Nor does she want a KPI that changes with every refresh of the dashboard.

A Semantic Layer has the potential to bring enough determinism to your LLM deployments to make it useful in the context of enterprise analytics. More on that and what we are doing about it at Strata in future posts.

Subscribe now

Recipe for Success

Okay, so you’ve bought in, found the right users, and the right use cases. You know how to avoid extreme adoption. Whats next?

Analytics Stack

We need to first understand our Analytics stack. You should think of it as a funnel where you have the most users with shallow usage at the top and the bottom will have few users but intense usage. It might look something like this:

Every organization is different and you might have some differences. This captures an average view of enterprises. The width represents the number of users and the depth represents the intensity (i.e. use it several times a day vs once a month).

Notice the red above. This is where you need to slot in the Semantic Layer. Less users than Executive View consumers, but very intense. I should say it has the potential for intense usage given a good Semantic Layer powered self service tool. Now that we’ve identified where in our stack it fits and for what use case, we need to build a data model.

Data Modeling

This will be your first stumbling block. Don’t let endless debates over building the perfect enterprise model derail your mission. The mission being to enable deep self service access to a specific set of users in a specific domain. Look I would love an enterprise data model with all the bells and whistles too. But, you can get there iteratively without boiling the ocean all at once.

How good the data model is will depend on your Semantic Layer technology. Some are quite dumb and will need a lot of help and others are more forgiving. We are building the latter at Strata but with some developer optimizations to grow with the team.

We’ll post more on this topic in the future. So subscribe!

A Curator with Taste

It was never going to be that easy. This isn’t absolutely necessary but it will really unlock value in ways you can’t understand until you see it in action. This person doesn’t have to do this as full time job. The initial development will be intense but after that it can be maintained on a part time basis. They can be one of the domain experts who partner with your Data Engineers, or a Data Engineer. Nonetheless, you need someone to be the lead curator.

But, why?

The Semantic Layer can be described as labels that point to data assets. These labels can be created haphazardly or with deep empathy. You have to understand how the users in the domain think, what is important, and how the Semantic Layer will impact their workflow. Some data will need to be cached because its used often while others can stay in deep storage.

Think of this role as your editor. Everything in your data model isn’t useful and many things need to be renamed to fit the mental model of your target users.

With that final piece in place you are now ready to unlock the true value of Semantic Layers.

Summary

Congratulations on reading this far! Your going to be among the leaders who run successful analytics programs. By now you should understand how to screw up a Semantic Layer deployment and also how to succeed.

I’d love to hear your comments or feedback. Will try to answer as many as I can. In the mean don’t forget to subscribe. Strata is building the next generation of Self Service reporting and analytics with Semantic Layers.

After another stint at Netflix, this one longer than anticipated at 8 years, I quit. Again.

I’m starting up a new business in a space I’ve been obsessed with for over 20 years now. Why am I obsessed with Semantic Layers and their ability to maximize analyst happiness? Doesn’t matter. I guess I grew to love what I’m good at.

We have lots to discuss in this new era of AI. Follow along as we build a new B2B business in the Business Intelligence space. This is not going to be the usual startup blog slop. Get the good, bad, and ugly. I’ve been a founder before and it’s not all fun and games. Subscribe and follow our journey.

Subscribe now

This Time is Different

This will be my third time building a Semantic Layer from scratch. But this time I think I truly cracked it. It just happens that my last two and half years at Netflix was spent honing in on the right product. This was my second attempt and I was able to resolve the many issues with off the shelf BI tools like Looker, MicroStrategy, Tableau etc when it comes to self service.

Some people try to deploy Semantic Layer technologies as a panacea when the right approach is to target a much more narrow use case. I will say more on this in future posts.

Why I Left Netflix

If you’re in the tech biz working for any of the big names, you know Netflix’s has the golden handcuffs of golden handcuffs. (maybe Meta now for AI stuff) Needless to say it’s extremely hard to give all that up for no salary :). You have to be a little crazy.

Anyways, as I alluded earlier I did build a BI tool with a powerful Semantic Layer at Netflix. It is currently deployed in production for several projects. In fact, I kicked off a sort of internal race to build Semantically enabled solutions. When I started working on it people were just getting reintroduced to Semantic BI tooling. In fact, it was once the default way for analysts to access data at Netflix. But, abandoned due to many reasons worth a post of its on. Long story short, the leading offerings were too hard to implement and maintain as the company grew.

Back to the main plot. I made a wild choice to win the race. I went against the standard nodejs, python, or java stack and chose Ruby on Rails. It was once popular at Netflix but now banned. Persona non grata!

Despite the end users loving and adopting the tool faster than ever, its success was being politically limited. Teams of many more developers were pushing promises while my solo developed solution was already in production. My choice of Ruby on Rails being off the paved path was the argument against further adoption.

It is a fair argument but we all need to veer of paved paths at some points to discover new possibilities. With the the Ruby on Rails renaissance we are seeing, I’ll be proven right for eventually. Nevertheless, this was also my cue to leave.

On Startups

Startups are hard. Startups are fun. But, Startups are hard.

This time I’m going to launch an LLC and delay funding til I’m satisfied that we are at or near product market fit. Maybe we’ll never raise. We are in a new tech epoch with AI. Lots to learn and innovate in terms of how you build a business.

I’m talking to several potential cofounders. Almost had a person quit and join me for no money just equity only to vanish suddenly. The ride starts immediately and before you know it your going down and then up. It’s thrilling!

With AI maybe Solo is the right move.

Why Ruby on Rails

If you made it this far, you probably guessed I love Ruby on Rails. My philosophy in building data tools for the past 20 years aligns well with Rails philosophy. Namely “complexity compression”. Having worked on internal tools at large enterprises it feels like complexity in every domain is increasing. Whether its deployment tools, data tools, web development frameworks, it doesn’t matter. In fact, I think large corporate management teams are incentivized towards complexity. Even with AI they will chose the complicated solutions over the simple ones at every turn. I bet there is a Dilbert comic strip showcasing this.

In anycase, Ruby on Rails since version 7 is experiencing a well deserved renaissance. Hotwire with Stimulus is a great answer to building modern Web UI’s. Even still you can easily add React or some other frontend framework. I think we will see many more startups choosing Ruby on Rails in the coming years.

At Netflix, I was just a bit ahead of trend. In time, my former colleagues will come around.