This and upcoming three texts could be a little bit more technical and less funny but hey - we need to go through this stuff. I tried to describe it in a way that is as accessible as possible for non-technical people.
Disclaimer: The Feature Store is one of the concepts of transformation tools for large organisations. The following description may contain details of the solution that will not fit every company.
TL;DR: Large organisations – from banks to retailers to technology companies – are drowning in data definitions that are created simultaneously by dozens of teams. Feature Store (FS) is the answer to this chaos: it is a central “feature catalogue” with a machine that calculates, versions and shares them.
One definition, one point of truth, one audit. The results? Fewer conflicting reports, shorter model deployment times, better regulatory compliance and lower cloud costs.
Problem: chaos in a large company
In a corporation, each department looks at data through its own lens.
Marketing counts “customer activity” as the number of logins to the application, the sales team counts it as the number of transactions, and the risk department also takes fraud alerts into account.
Without a common vocabulary, three similar but incompatible columns are created. When they are combined into a single model, the results are a lottery, and it is difficult to identify who is at fault. I’m sure you have your own examples from your company.
The problem escalates when hundreds of analysts copy and modify each other’s notebooks/reports: technology debt and processing costs quickly multiply.
In addition, a regulatory audit can block a project for months if the organisation is unable to reconstruct the full data path.
Basic concepts (for non-technical users)
^ See what I did there? You really, really need to read this.
- Feature – a single number or label that the model understands, e.g. “how many GB of data the customer has used in 7 days”.
- Feature set – a group of such numbers relating to the same object (e.g. customer number).
- Offline Store – a historical archive, cheap and slower; ideal for training models.
- Online Store – a fast cache of current values; crucial for real-time decisions.
- Point‑in‑time – the principle that we only use information available at a given moment when training (the model cannot “see the future”).
Imagine FS as a “Wikipedia for features”: each article has a single owner, a clear definition and an edit history.
Why it’s worth it (superpowers)
- One language in the company – no more emails asking “which version of ARPU is correct?”.
- Consistency between laboratories and production – the same code calculates features in the warehouse and in the live system.
- Audit and regulations – full reproducibility: proof that the credit decision was based on specific data.
- Cloud savings – no duplication of calculations and transfers.
- Faster experiments – ready-made features reduce prototyping from weeks to hours.
Reference architecture + BFF (Backend for Frontend)
The system consists of four layers:
- Ingestion – batch processes (e.g. nightly data warehouse feeds) and streams (events from applications, IoT, transaction logs).
- Transformations – code (e.g. SQL/Python) converts raw records into useful features; this is where schema validation and quality testing take place.
- Stores – Offline Store for history, Online Store for the last minutes or hours of data.
- Serving API – feature retrieval point.
Since there are many client channels (mobile app, IVR, portal, chatbot), thin BFF microservices are often added.
Each BFF trims the response from the FS to the needs of a specific frontend: the mobile app downloads features in batches and caches them on the phone, the chatbot needs fast JSON, and the web portal wants additional flags for the advertising module.
BFFs provide interface flexibility without breaking the uniform FS core.
Feature lifecycle (again, for non-technical users)
- Idea – someone from the business says: “Let’s check how many customers contacted the helpdesk in 48 hours.”
- Design – a data engineer records the definition (name, source, refresh frequency) in a form.
- Construction – an automated tool generates code that extracts logs and counts the specified metric.
- Testing – the system checks whether the result makes sense (e.g. number ≥ 0) and whether it refreshes on time.
- Publication – the feature is added to the catalogue; from this moment on, any model can use it.
- Monitoring – the dashboard tracks the freshness and distribution of values.
- Retirement – a feature that has not been used for 90 days ends up in the queue for deletion
Kill the data you don’t use.
I know how tempting it is in corporations to store data “just in case,” but please, make sure you have rules for deleting unused features and stick to them.
Another argument large organisations use against removing features is “b-b-but we paid for it; there was a project that implemented it.”
Well, no organisation is large or rich enough to afford to maintain a dumping ground for abandoned data. If you have to store it (e.g. for legal reasons) - archive it for the required time and don’t add new data. Kill the feature that provides it.
Batch vs Streaming – two paths to the truth
Batch is a classic nightly data package. It provides stability, low cost, and the ability to reprocess the entire history. It works great for features based on long windows: average monthly basket, number of logins per quarter, or sales trend regression.
Engineers value batch for its ease of debugging: if something goes wrong, just restart the task and compare the two versions of the file.
Streaming is the other side of the coin: data flows in a continuous stream and the feature is updated second by second. This is useful when a decision has to be made immediately (anti-fraud systems, dynamic pricing, real-time personalisation).
However, building a stream requires discipline: you have to handle late events, set time windows and monitor delays (watermarks). The infrastructure cost is likely to be higher because the streaming engine runs 24/7.
In practice, companies choose a hybrid solution.
The last hour or day of data travels in a stream – where instant response is crucial. Older history ends up in a batch. This allows SLAs to be met in seconds where it really matters, without excessive infrastructure spending.
FS allows both paths to be described with a single specification: the processing_mode = AUTO parameter selects the strategy based on freshness requirements. When the business team changes the SLA (e.g. from “every hour” to “every 5 minutes”), all it takes is a single line modification in the registry, and the orchestrator will generate a new DAG, keeping the business definition intact.
processing_mode = AUTO is not an “official” parameter of a single, specific tool, but a short name for a concept.
Equally important, batch and streaming can be combined in a “lambda-less” mode – the same transformation logic works on two runtimes, so there is no definition drift.
This approach simplifies maintenance: unit tests and schema validations are shared, and the DevOps team maintains a single set of artefacts.
“Lambda-less” is simply an agreement between the batch and streaming worlds: you write the logic once, and the framework or storage layer makes sure it works for history and streaming. In the context of Feature Store, this simplifies feature management, minimises train/serve discrepancies, and reduces operational debt – at the cost of a slightly more demanding platform underneath, of course.
Versioning and time travel
In FS, versioning is similar to a code control system: each change to a feature definition is assigned a number (e.g. v2.0.0). The difference is that we version not only the text, but also the result of calculations.
When an engineer corrects a filtering error – e.g. excludes duplicate transactions – a new version of the feature is created. The old one remains intact, and the system stores a map that says: “models A and B use version 1.x, model C uses version 2.x”. This ensures that migrations are controlled and models do not stop working overnight.
Time travel is a feature that allows you to ask: “Show the set of features for user X at 17:07 on 14 February 2025.” FS bases this on snapshots of the offline store partition.
Implementation requires a copy-on-write (separate delta files) or merge-on-read (change log + background compaction) strategy. In large organisations, 6-18 months of history is stored on a fast drive, and older versions are compressed or moved to a cheaper layer (e.g. an archive in an object).
An additional value is the semantic diff feature. Registry can generate a report comparing the distribution statistics between two versions of a feature. If the number of non-zero values drops by 90%, this is an alarm signal that the filtering has gone too far.
Such tools shorten model regression debugging: instead of looking for a needle in a haystack, the analyst can immediately see which variable has dramatically changed its character.
Versioning also supports A/B experiments
A new version of a feature can be “split” into a test branch and served only to a portion of traffic. If product metrics improve, the branch is merged and marked as stable. All this without manual code copying and manual deployment.
Security and compliance
Regulated organisations – banks, healthcare, telco, public sector – need to know who has viewed specific data and when. FS provides several levels of protection:
- Confidentiality labels – each column has a tag (low, high, PII). Any attempt to read PII data by a user without the appropriate role is automatically blocked.
- End-to-end encryption – data is encrypted at rest (disks, objects) and in transit (TLS). KMS keys rotate every 24 hours and access is auditable.
- Right to be forgotten – FS stores a data location map for each ID. When a deletion request is received, the system launches a process that atomically deletes all versions of the record, generates a proof hash and writes it to an audit log (WORM – write once, read many).
- Access reports – The “who queried feature X in the last quarter” API returns the total number of calls, a list of users, and the time. This is crucial for compliance checks.
- Client-side policy enforcement – the FS library injects tokens into every query. If the token does not have the appropriate permissions, the query is rejected at the SDK stage. This ensures that data does not leave the controlled channel.
To extend protection, many companies add a layer of differential privacy
FS can return noisy statistics for ad hoc exploration, and access to raw values requires a separate request path. As a result, novice analysts can test hypotheses without the risk of information leakage, and productive researchers receive full data only after the project has been reviewed.
Pitfalls and lessons learned
- Overly enthusiastic start – a full-scale FS without a clearly defined list of priority features ends in paralysis. Start with the top 5 variables that affect key business metrics.
- No catalogue owner – in a large company, responsibility can be blurred. Appoint feature product owners and rotate shifts as in SRE.
- Over-reliance on a single team – if only the ML department understands FS, the rest of the organisation will revert to Excel. Educate and provide clear documentation.
- Ignoring the budget – every new feature comes with storage and computing costs. Set “entry” rules and freshness limits.
Disadvantages and challenges
Feature Store is not a silver bullet – before you throw your entire team into it, it’s worth knowing the darker side:
- Complexity of implementation – FS introduces an additional layer of infrastructure. It takes time to understand the new data lifecycle and synchronise it with existing data warehouses and orchestrators.
- Cost of expertise – maintaining a feature store requires engineers who understand both data and cloud operations. This talent is still expensive on the market.
- Vendor lock-in risk – commercial platforms are tempting because of their convenience, but they can make data migration difficult if you change cloud providers or strategies. If you don’t have the budget for your own solution, choose the platform-agnostic approach for commercial ones.
- Single point of failure – by centralising features, you create a critical axis. An FS failure can stop scoring in multiple systems simultaneously. You need to invest in high availability and a contingency plan.
- Excessive formalities – every new feature goes through a review process; this is effective, but can slow down innovation if the procedures are too rigid.
- The “golden hammer” effect – the temptation to pack everything into FS, including ad hoc analyses, which increases clutter and costs.
The key is balance.
- Start with clear goals,
- build a minimal, stable core,
- and expand only when the benefits outweigh the overhead.
When I started writing The Rise of Ops, I wanted to talk about how operationalisation (as part of digital transformation) is changing the way companies work, especially at the corporate level.
Feature Store is one of the milestones of this transformation: it takes data logic from chaotic spreadsheets and notebooks and puts it into a manageable, versioned system. It unlocks innovation by allowing engineers to focus on ideas rather than searching for differences in SQL. It ensures compliance without losing agility. And most importantly, it builds a common vocabulary between technology and business.
Three more toys to talk about.