Snowflake
Snowpark: DataFrame API & ML Workloads
Write Python/Java/Scala data pipelines and ML workflows that execute inside Snowflake's compute.
Snowpark lets you write transformations in Python, Java or Scala, and runs them on Snowflake's own warehouses.
Snowpark is like writing your data analysis in a familiar notebook language, but every operation you write is secretly translated into instructions handed to the same powerful warehouse crew that already runs your SQL, so the data never has to leave the building to be processed.
Key Concepts
1
The point is where the computation happens. Normally a Python pipeline pulls data out of the warehouse,
processes it on another machine, and writes results back — so a large table has to cross the network twice
and the other machine must be big enough to hold it.
2
Snowpark translates your code into SQL and runs it where the data already is.
3
The DataFrame is lazy, and that is the mechanism.
df = session.table("orders") \
.filter(col("status") == "PAID") \
.group_by("customer_id") \
.agg(sum_("total").alias("spend"))
df.show() # only now does anything run4
Each call builds up a plan. Nothing executes until an action — show(), collect(),
write.save_as_table() — at which point the whole chain becomes one SQL statement.
show()collect()write.save_as_table()
5
So the filter and the aggregation happen inside Snowflake, and only the final rows come back. If it
executed eagerly, the first line would pull the entire table to your client.
6
Familiar shape, different execution. The API is deliberately close to Spark and pandas, so the
ergonomics transfer. What does not transfer is the mental model of data living in your process — here it
never does, which is why collect() on a huge DataFrame is the mistake to avoid.
collect()
7
Snowpark ML keeps training next to the data. A scikit-learn-compatible API for preprocessing and
training that runs distributed inside warehouses, plus a model registry and deployment as a UDF:
8
SELECT id, predict_churn(features) FROM customers;9
Inference becomes a column in a SQL query. The model runs inside Snowflake, so scoring a large table does
not export it anywhere.
10
The governance argument is the real one in a regulated setting: the data never leaves, so the access
controls, masking and row policies still apply during training and inference.
11
Python UDFs and procedures use the same machinery. Custom logic runs in a sandbox, with third-party
packages available from Snowflake's curated Anaconda channel. That curation is also the limit — a package
outside it needs to be uploaded as a stage file, and some are not usable at all.
12
When it is the wrong tool. If the transformation is naturally expressible in SQL, write SQL: it is
simpler to read, easier for others to maintain, and runs identically. Snowpark earns its place where the
logic is genuinely procedural, where a Python library is needed, or where an existing Spark pipeline is
being moved without a rewrite.
13
What the interviewer is probing.1. "What is the point of Snowpark?" Probing: the value. Stalls: "Python support." *Moves
up:* the DataFrame operations compile to SQL and run in Snowflake, so data is not pulled to a
client.
14
2. "When is it lazy?" Probing: the model. Stalls: "Never." Moves up: transformations build
a plan; nothing executes until an action like collect, show or write.
15
3. "What is a vectorised UDF for?" Probing: the performance detail. Stalls: "Speed." *Moves
up:* processing batches as pandas Series rather than row by row, which is far faster for Python
logic.
16
4. "What runs where?" Probing: the mental model. Stalls: "All in Python." Moves up:
DataFrame work becomes SQL on a warehouse; UDFs and stored procedures run in a sandboxed Python
runtime inside Snowflake.