Is your data actually ready for AI?
Probably, for most of what you want to do, and the question is asked far more often than it needs to be. A great deal of useful AI integration involves no training data whatsoever, because a general-purpose model is reading a document you already have. Data readiness only becomes the deciding question when you want a model to learn your specific patterns, and knowing which situation you are in saves months.
Which problems need no data at all?
Anything where the intelligence required is general rather than specific to you. Summarising a support ticket, extracting fields from an invoice, drafting a reply, classifying a message by topic, translating, or answering a question from a document you supply at the time. A general model already does these, and your data is the input rather than the training material.
This covers the majority of what companies actually want, which is why the readiness question is often a distraction. If your project is in this category, the relevant preparation is access rather than quality: can the system reach the documents, and is anyone allowed to send them to it.
The line to test for is whether the answer depends on your history. Extracting a total from an invoice does not. Predicting which invoices will be paid late does, because that pattern only exists in your own records.
When does your own data become essential?
When you want prediction, ranking, or a judgement that reflects how your business specifically behaves. Forecasting demand, scoring leads, predicting churn, flagging anomalous transactions, recommending the next product. None of these can be answered by general knowledge, because the answer is a property of your customers rather than of the world.
It also becomes essential when you want a model to adopt your vocabulary and conventions closely enough that generic output is unacceptable, though this is a smaller category than people assume and often solved by retrieval and good prompting rather than by training.
For the prediction cases the requirement is specific and worth stating plainly: you need history, and you need the outcome recorded. A dataset of ten thousand customers is useless for churn prediction if nobody wrote down which ones left.
| What you want | Needs your data? | What actually matters |
|---|---|---|
| Summarise, draft, translate | No | Access and permission, not quality |
| Extract fields from documents | No | A consistent document format |
| Answer questions from your docs | No, retrieval not training | Documents being current and findable |
| Classify into your own categories | A little | A few dozen labelled examples per category |
| Predict churn, demand, or risk | Yes | History with outcomes recorded |
| Recommend or rank | Yes | Logged behaviour, not just a catalogue |
What makes a dataset unusable?
Four things, and only one of them is size. Missing outcomes is the most common and the most fatal: if the thing you want to predict was never recorded, no amount of data volume substitutes, because the model has nothing to learn from. This is why churn projects stall at companies that never marked accounts as churned.
Leakage is the subtlest. If a field in your history was only filled in after the outcome was known, a model trained on it will look extraordinary in testing and fail completely in production. A cancellation-reason field is the classic example: it predicts cancellation perfectly and is only populated once someone has cancelled.
Then inconsistency, where the same thing was recorded three ways because three teams entered it differently, and drift, where the process changed partway through so the older half of the history describes a business that no longer exists. Both are fixable; both need to be found before anyone estimates the work.
How much data is enough?
Far less than the usual figures suggest, because the useful modern approach is rarely training from scratch. For classifying text into your own categories, a few dozen clear examples per category is often enough to be useful, and a few hundred is comfortable. For numerical prediction, the honest answer is that you need enough examples of the rare event, not enough rows overall.
That distinction is worth dwelling on. A hundred thousand transactions with forty confirmed frauds is a forty-example dataset for the purpose at hand, not a hundred-thousand-example one. Volume is measured in the thing you care about.
So the question to ask internally is not how many records exist. It is how many times the outcome you want to predict has actually happened and been recorded as such.
Does it need to be in a warehouse first?
No, and waiting for one is the most expensive form of delay in this field. Plenty of valuable integrations run against an operational database, an export, or an API, and a pilot does not need a platform behind it.
The warehouse argument is usually really an argument about repeatability: if the model needs fresh data daily and the only route is someone exporting a spreadsheet, that is a fragile arrangement worth fixing. But fixing it is a step you can take once the pilot has demonstrated the value, and doing it first means spending the budget before knowing whether the thing works.
A reasonable sequence is to prove the value on an extract, then invest in the pipeline for the specific data the working system needs, which is invariably a narrower set than a full warehousing programme would have delivered.
What should you check before commissioning anything?
Five questions, answerable in an afternoon by whoever knows your systems. Is the outcome we want to predict recorded anywhere, and since when? How many times has it occurred? Was any field in our history filled in only after the outcome was known? Has the process that generates this data changed in the period we hold? And who is allowed to approve this data leaving our network, if it needs to.
The last one is not a data question and it stops more projects than the other four combined, usually late, after work has been done. Establishing early whether documents can be sent to an external provider, and under what conditions, determines the architecture rather than being a detail to settle afterwards.
If those five have answers, the feasibility conversation with any supplier becomes short and specific, and you will be able to tell immediately which suppliers are listening.
Common questions
- Does AI integration always require training data?
- No. Most of what companies want, including summarising, drafting, translating, extracting fields from documents and answering questions from their own files, uses a general model where your data is the input rather than the training material. Your own historical data becomes essential only when the answer depends on how your business specifically behaves, such as predicting churn, forecasting demand or scoring leads.
- How much data do you need for a machine learning model?
- Fewer records than commonly assumed, because the count that matters is examples of the outcome, not total rows. A hundred thousand transactions containing forty confirmed frauds is a forty-example dataset for fraud detection. For classifying text into your own categories, a few dozen clear examples per category is often enough to be useful.
- What makes data unusable for AI?
- Four things: the outcome you want to predict was never recorded, which is fatal; leakage, where a field was only populated after the outcome was known, producing a model that tests brilliantly and fails in production; inconsistent recording across teams; and process drift, where the older history describes a business that has since changed. Only the last two are straightforwardly fixable.
- Do we need a data warehouse before starting an AI project?
- No, and waiting for one is the most expensive way to delay. A pilot can run against an operational database, an export or an API. Build the pipeline afterwards, for the specific data the working system turns out to need, which is always a narrower set than a full warehousing programme would deliver.
- What is data leakage in machine learning?
- When a field used to train a model was only filled in after the outcome it is meant to predict was already known. A cancellation-reason field predicts cancellation perfectly and exists only once someone has cancelled. Models trained on leaked fields score extraordinarily well in testing and fail immediately in production, which makes leakage the most dangerous data problem because it looks like success.
