Strip away the scale and the chat interface, and a language model is one idea: turn the world into sequences of tokens, then train a network to predict the next one. The idea never cared that the tokens were words. This essay walks through what happens when the tokens are transactions, orders, and claims instead, the technical core of an enterprise foundation model.
Events are sentences
Consider one checking account for one month. A payroll deposit. Rent, two days later. Groceries on a weekly rhythm, coffee on a daily one. A transfer to savings sized to what's left over. This is not a bag of independent rows, it's a sequence with dependencies at every range. The rent follows the deposit. The savings transfer depends on the whole month. A sudden burst of unfamiliar merchants depends on nothing that came before, which is exactly what makes it interesting.
Local structure, long-range structure, and meaningful violations of both: that is the profile of natural language, and it's why the same architecture applies. One customer's history reads like a sentence. A company's history reads like a corpus.
Tokenizing structured events
The first engineering problem is representation. A word is one symbol; an
event is a typed record, type, amount,
merchant, channel, timestamp, and a
dozen other fields. Three design decisions matter:
- Fields become sub-tokens. Each event expands into a short phrase of tokens drawn from learned vocabularies, categorical fields map directly, high-cardinality fields (merchants, SKUs) get their own embeddings, and continuous values like amounts are quantized into buckets so the model can treat magnitude as vocabulary rather than regress on raw floats.
- Time is a first-class signal. Behavioral data has no fixed tempo, the gap between events carries as much information as the events themselves. Encoding inter-event time (and calendar position: day of week, day of month) lets the model learn rhythms and, crucially, notice when they break.
- Sequences are per-entity. The corpus is organized as one sequence per customer or account, so attention operates over an entity's actual history rather than an arbitrary shuffle of rows. This is what makes the learned representation an embedding of the customer, not of a transaction.
Note what is absent: feature engineering. No "average spend over 90 days," no "days since last purchase." Those aggregates are lossy summaries invented so small models could cope. A sequence model reads the raw history, so nothing is thrown away for it to recover later.
The objective: predict what happens next
Training is autoregressive next-event prediction: given the sequence so far, output a distribution over what comes next, its type, its attributes, its timing. The loss is cross-entropy over the factored event, summed over every position in every customer's history. The labels are the data itself; no human annotates anything.
The reason this humble objective produces a foundation model is the same reason it worked for language: the task is only easy if you understand everything. To place probability mass well, the model is forced to infer the customer's segment (a student's sequence continues differently from a retiree's), their latent state (recently unemployed, newly a parent, shopping for a mortgage), product relationships, seasonality, and risk dynamics. Those inferences have nowhere to live except the network's representations, which is precisely what we're after.
Pretraining doesn't ask the model any business question. It asks the one question whose answer requires knowing the answers to all of them.
What the representations hold
After pretraining, the model's hidden state at the end of a customer's sequence is a vector summarizing everything predictively relevant about them. Probing these embeddings shows linearly-recoverable structure the model was never told about: tenure, affluence, life-stage, engagement trajectory. Clusters correspond to segments a marketing team would recognize, and to some they wouldn't, because no one hand-designed them.
These embeddings are the interface between the foundation model and the business:
- Fine-tuned heads. A small classifier on top of the embedding, trained on modest labels, becomes a churn model, a credit-risk model, a propensity model, in days, because the hard representational work is already done.
- Anomaly scores for free. The pretraining objective itself is a fraud detector: events the model assigns low probability are events that don't fit this customer's grammar. No labeled fraud required to start.
- Retrieval and lookalikes. Nearest neighbors in embedding space power recommendations, audience expansion, and cold-start reasoning about new products.
Practical notes from the field
Scale is enterprise-sized, not internet-sized
A useful pretraining corpus is millions to billions of events, what a mid-size bank, retailer, or insurer already has in its warehouse. Model sizes follow the data: these are not trillion-parameter runs, and a full pretraining fits comfortably inside a single company's cloud budget, which is also what makes training inside the customer's VPC feasible rather than aspirational.
Evaluation is held-out future, not held-out rows
The honest split for behavioral models is temporal: train on history up to a cutoff, evaluate on what actually happened after. Random row splits leak the future into training and flatter every metric. Next-event likelihood on held-out time is the model's perplexity; downstream AUC and lead time on frozen embeddings measure what the business gets.
Drift is a fine-tune, not a rebuild
Businesses change, new products, new fraud patterns, new seasons. Because the foundation model is self-supervised, refreshing it is a continued pretraining pass on recent events, and every downstream head inherits the update. Compare that to the feature-pipeline world, where drift means re-engineering each model separately.
The takeaway
None of the machinery above is exotic, tokenization, attention, autoregressive loss, fine-tuning. That's the point. The transformer recipe is the most validated in the history of machine learning; what's new is aiming it at the sequences enterprises actually run on. The result is one model that has read every event in your history and can be asked, cheaply and repeatedly, what your business will do next.