Data Flows: Building Reusable Data Pipelines in OAC

Explain how OAC Data Flows let users build reusable, multi-step data transformation and machine learning pipelines without writing ETL code.

A data flow is a visual pipeline producing a new dataset. It is the step up from one-off preparation to something repeatable that other projects consume.

A Data Flow is like a home assembly-line kit: you snap together pre-built stations (filter, join, calculate, predict) in whatever order you need, and once assembled, the same conveyor belt runs automatically every time new raw material arrives — without needing an engineer to redesign the factory floor each time.

Key Concepts

A flow is a sequence of steps on a canvas.

    add data       source a dataset or a subject area
    filter         keep the rows you want
    join           combine two sources
    add columns    calculations
    aggregate      group and summarise
    train model    fit a machine learning model
    save data      write the result as a new dataset

The difference from preparation. A preparation script belongs to one dataset and shapes it. A data flow takes several inputs, transforms them, and produces an output other people use.

    preparation   this file, cleaned, for me
    data flow     these sources, combined, as a dataset for everyone

It can be scheduled, which is what makes it a pipeline rather than a one-off — run nightly, refresh the output dataset, and every project using it sees the new data.

Where it sits between the alternatives.

    ODI or a warehouse ETL   governed, versioned, tested, owned by a
                             data team. Slow to change.
    a data flow              built by an analyst in an afternoon.
                             No deployment, no ticket.
    blending in a project    ad hoc, inside one project, not reusable

Data flows fill a real gap: the transformation that is too involved to repeat by hand and not important enough to justify a warehouse change request.

And that is also the risk. A data flow producing a number the business relies on is a production pipeline that lives in a BI tool, built by one person, with no tests and no version history.

The honest position is that it is right for analysis and prototyping, and that anything it produces which becomes load-bearing should be promoted into the managed pipeline.

Machine learning steps are available inside a flow — train a model on a dataset, then apply it to score new data — which is what makes the augmented analytics features usable without a separate tool.

The practical caution. A flow reading a large source runs the whole transformation each time it executes. It is not incremental, so a nightly flow over a growing table takes longer every night until somebody notices.

What the interviewer is probing.1. "What is a data flow for?" Probing: the purpose. Stalls: "Loading." Moves up: a visual sequence of transformations producing a saved dataset, so preparation is repeatable rather than redone per workbook.

2. "What can it do?" Probing: breadth. Stalls: "Filter." Moves up: join, aggregate, filter, add columns, apply a machine learning model, and output to a dataset or a database table.

3. "When is a data flow the wrong tool?" Probing: judgement. Stalls: "Never." Moves up: for enterprise ETL at volume with dependencies and restartability — that belongs in ODI or a proper pipeline.

4. "How is it scheduled?" Probing: the operational detail. Stalls: "It is not." Moves up: as a sequence with a schedule, though the orchestration features are deliberately lighter than a dedicated ETL tool.