'Dark data' is your new most valuable data for AI

Someone asked me a question the other day related to generative AI (because why not AI), and it went something like ‘what do you think is going to be the next trend here in 6 to 12 months?’.

The simplest answer I have is that big data is coming back, and it’s going to be highly relevant to many teams as the agentic AI era continues. But the foundation of agents isn’t built on your structured data. It’s built on logs, interactions, and the mess of data residing in your Google Workspace, Office (CoPilot365), and other platforms that have been either off limits, too expensive to track, or frankly, just not considered as valuable.

A ‘simple’ CRM agent example

Here’s a simple business case. Let’s say you are going to create a chatbot for a sales agent. The agent is designed to be an assistant for your sales team to answer questions about specific deals, deal designs, customer analysis, and even deal support. You probably have a CRM that has the basics like what the customer account is, some sloppy opportunity data, and some light ‘contact’ information.

But the information your agent needs in order to learn is all about the interactions that take place across prospects, deals, and opportunities. This data might look something like:

• Meeting transcripts between a sales representative and a prospect. Other transcripts between your internal teams talking about the deal size, approach, and even contracting.

• Spreadsheets with analytics in them about the deal and different values.

• Log data from your other systems (say your actual CRM logs) for changes to the deal over time, or how a what-if scenario engine is used.

For most organizations, this represents a pile of very messy, large data. And frankly, it wasn’t that useful before. However, agents need to understand the glue (aka processes) that actually create outcomes in order to re-enact and improve. Imagine you have to improve one of your skills, and the only feedback you have is the input, and the outcome. Aligning your own trajectory to the outcome is a hit or miss activity, and where we typically say the agent failed and move on when we didn’t give the agent anything useful to get started with.

This is the ‘dark data’ problem in companies, and where big data, in my opinion, is going to make a comeback.

Building an agentic ‘big’ data foundation

So what do data and AI leaders do to get ahead of this?

1. Get ahead of this problem and hoard as much telemetry and unstructured data as possible. I know we say hoarding creates data swamps, but when change is constant, understanding what in this data is useful is unknown, the compute power these days can sift through it, and not having eval data means your agent is DOA.

2. Get value stream maps of the processes you are looking to ‘agentify’ and map your data domains to this. If you don’t have the data today, identify all the sources you might have available including (gasp: new SaaS systems) that can get this process data for you.

3. Start identifying golden data outcomes from your captured data. This may be something like ‘sessions that yielded a sale’ or ‘key process moments’. Labelling data provides you with an eval dataset that can be used when training or modifying agent behavior to yield similar agent trajectories (think an agent session with turns and tool calls).

There are many other things here that you’ll need like managing sensitive data effectively from the data governance playbook, and building effective and efficient data models over this, but step 1 is still ‘collect data’.

Happy Wednesday!

$ uv run python -m electric_sheep.dream --scope workspace --min-confidence 0.90
DREAM CONSOLIDATION COMPLETE · Scanned 18 session trajectories
PROMOTED BELIEF #042: 'Always run uv run pytest -v and maintain <3s unit test execution' (confidence=0.97)
SPANNER COMMIT: projects/davenport-boutique/instances/electric-sheep/databases/memory