Code Execution with Gemini

I want to tell you about a small but powerful idea: connecting a large language model to a tool that actually runs code. I dug into the code execution tool for Gemini and built a couple of demos that show how this changes the game for math, data analysis, and code-based reasoning. The short version is simple: language models are great at language, and very often they are also great at writing code. If you let them write code and then run that code in a controlled environment, you can get correct numeric answers, reproducible data analysis, and clear evidence for how the model arrived at a result.

In this article I report on what I tried, what worked, what failed, and how you can use the code execution tool in your own projects. I wrote this as a step-by-step news-style walkthrough mixed with practical advice, and I include a FAQ at the end to answer the most common questions I see people ask.

🔎 What I tested and why it matters
🧮 The first demo: summing the first 60 prime numbers
📊 A practical app: CSV Genius for data questions
🛠️ How the tool is wired (in broad strokes)
🔐 Safety, security, and sandboxing considerations
⚙️ Best practices for prompting with code execution
🔁 Reproducibility and debugging
🧭 When to use code execution and when not to
📈 Real-world use cases I imagine
🔍 A deeper look at the technical trade-offs
✍️ Prompt examples I used and why they work
💡 UI and UX tips for apps that expose code execution
🔬 Common failure modes and how I handled them
🔗 Integrations with other tools and APIs
🧾 Example of a complete end-to-end scenario
🔭 Where this tech is heading
🔁 Closing thoughts and quick tips
❓ Frequently Asked Questions
📣 Final notes

🔎 What I tested and why it matters

Large language models, including Gemini, are trained to predict the most probable next token in a sequence. That mechanism is the root of their power for language tasks, but it is not optimized for exact numerical computation. In plain language, LLMs are not calculators. They can hallucinate numbers, approximate results, or produce different numeric answers on different runs.

I wanted to explore how code execution changes that. The idea is straightforward: ask the model to produce Python code that solves the problem, then execute that Python code in a sandboxed environment. The model still does the reasoning and code generation, but the numeric computation is done by the Python runtime, which is deterministic and precise for tasks like summing integers, running statistical computations, or manipulating CSV data.

The key benefits I set out to show are these:

Accuracy — reduce numeric errors for math problems by delegating computation to an interpreter.
Transparency — generate and inspect the code that produced the answer, which helps debugging and trust.
Productivity — let people who do not know pandas or Python ask questions about data and get reliable answers.
Reproducibility — run code on a backend so answers can be reproduced and audited later.

🧮 The first demo: summing the first 60 prime numbers

I started with a neat little math test: "Calculate the sum of the first 60 prime numbers." It's a clear goal, deterministic, and small enough to reason about. When I first asked Gemini without any tools enabled, I saw different numeric answers on repeated runs. On one run the model reported 8,368. That number is wrong. The correct answer, after I checked independently, is 7,699.

Why did the model get it wrong sometimes? Because without a dedicated calculator or code execution, the model is trying to produce a numeric answer token by token using learned patterns. For many tasks that works well, but exact integer sequences and sums are brittle because the model might not have reliably memorized that particular sum or might be approximating steps in its reasoning.

Then I turned on the code execution tool. I kept the same prompt but toggled the tool that lets the model return Python code that is executed on the back end. The model generated a short Python function to test primality and then iterated over integers to add up the first 60 primes. The Python runtime executed that code and returned 7,699. Success.

Two important observations from that test:

When code execution is enabled the model often returns runnable code instead of a long textual reasoning chain. That code is precise and auditable.
The numeric computation itself is handled by Python, avoiding the unreliability of token-by-token numeric generation.

This pattern is the simplest demonstration of the new workflow: ask for an answer, but have the model provide and execute code to compute it. The result is both correct and traceable to the exact lines that produced the answer.

📊 A practical app: CSV Genius for data questions

I also built a tiny app I call CSV Genius. The app shows how useful code execution is for everyday data analysis tasks. I uploaded a CSV that represents sales transactions for a fictional tropical fruit stand. The file had columns like transaction id, date, product, quantity, and price. My goal was to let non-programmers ask natural language questions and get correct answers backed by code.

One question I asked the app was "what fruit did I sell the most of (quantity)?" The model returned "mangoes" and 63 units sold. That result is correct for the dataset I used. But as a responsible developer and journalist, I wanted to know where that answer came from.

With the code execution tool turned on the app not only showed the answer but also displayed the Python snippet the model generated and executed. The snippet used pandas-style operations: load the CSV into a DataFrame, optionally filter for fresh fruit, group by the product column and sum the quantity, then pick the top product. That exact code was run on the back end and produced the numeric output returned to me.

Having the generated code visible is crucial for trust. If you see a surprising result, you can inspect the code, rerun it locally, or tweak the filters. Unlike a simple language-only reply, you get both the explanation and the exact program that produced the answer.

🛠️ How the tool is wired (in broad strokes)

When I built the CSV Genius app I used Google AI Studio with the Gemini model set to its Flash variant. In the app code I call the AI chat creation method and pass in the model identifier along with a parameter that tells the service to enable code execution. Internally the LLM returns a response that includes a chunk of Python. The platform sends that Python to a safe execution environment, captures standard output and return values, and then packages that back into the chat response.

From an application developer perspective the flow is:

User asks a question in natural language.
The app forwards the question plus the data context (for example, a CSV file) to the LLM with code execution enabled.
The LLM returns Python that attempts to answer the question using the data it can access.
The execution environment runs the Python code and returns the results to the LLM or directly to the app.
The app displays the result and optionally shows the executed code for inspection.

This pattern decouples the model's reasoning from the numeric computation and gives you a chance to audit the steps. In my CSV demo I show both the final answer and the code that created it. In the prime-sum demo I get the exact numeric output from Python rather than a probabilistic language guess.

🔐 Safety, security, and sandboxing considerations

Whenever you let models generate and run code, safety is the top concern. I treated the execution environment as a sandbox, and you should too. Here are the things I paid attention to and that you need to consider if you build something similar.

Filesystem access — limit or prevent writes to persistent storage. Allowing arbitrary writes can lead to exfiltration or persistence of malicious artifacts.
Network access — by default, block outgoing network calls that the generated code might attempt. If you do enable network access, limit destinations and monitor traffic carefully.
Resource limits — impose CPU timeouts, memory caps, and runtime duration limits. Generated code can loop forever or exhaust memory.
Package management — control which Python packages are available. Allowing arbitrary pip installs is a severe risk.
Input validation — never assume the model will only produce "good" code. Treat the output as untrusted and instrument the execution pipeline to keep the runtime safe.
Auditing — log executed code, inputs, outputs, and metadata so you can investigate surprising or malicious runs afterwards.

These are standard practices when exposing code execution. The novelty here is that the code is being generated dynamically by a model, so the execution environment still needs all those guardrails. In my experiments I used an environment that did not allow external network access and had strict CPU and memory limits. Even with those restrictions, the code execution tool was very useful for common math and data tasks.

⚙️ Best practices for prompting with code execution

To get reliable and auditable results you need to design prompts that encourage the model to produce clear, efficient code and to avoid dangerous constructs. Here are the rules I followed and would recommend to you.

Ask for runnable code only when you want computation. If the task requires exact computation, make it explicit: "Return runnable Python code that computes the numeric answer and nothing else."
Constrain output format. If you want JSON output or only the final numeric value, specify that. For example: "Return only a line with the final integer, nothing else."
Provide context and data access details. If the model will work with a CSV, explain the columns and mention file names or variables that are available in the environment.
Request code comments. Ask the model to include brief comments so humans can inspect purpose and reasoning quickly.
Prefer simple, deterministic approaches. For math tasks prefer straightforward loops or established library calls to reduce chance of subtle bugs.
Ask for tests. In more complex tasks, request a tiny self-check or assertion inside the code to validate assumptions about the data.

Example of a short, effective prompt I used: "Write runnable Python that loads 'transactions.csv' into a pandas DataFrame, filters for fresh fruit, groups by product, sums quantity, and prints the product with the highest quantity and the quantity number. Output only the product name and integer separated by a comma."

Prompts like that reduce ambiguity, produce concise, auditable code, and make it easy for the app to parse the results.

🔁 Reproducibility and debugging

One of the biggest wins with code execution is reproducibility. When the model returns code you can save that code, log the inputs, and rerun the computation later. That solves one of the fundamental issues with language-only answers: they are often not reproducible because the model may produce a different verbal reasoning chain on a different call.

When debugging I recommend this workflow:

Ask the model to generate the minimal code needed to answer the question.
Inspect the generated code before execution. Look for potentially expensive operations or network calls.
Execute the code inside a sandbox with monitoring turned on.
If the result is unexpected, edit the generated code or add logging statements and re-run the snippet locally to isolate the bug.
Persist the final code and inputs so you can reproduce the result exactly later.

Because the code is plain Python you can copy it into your own development environment and iterate. That makes the entire process feel like pair programming with the model: it drafts the code and you polish and verify it.

🧭 When to use code execution and when not to

Code execution is awesome for a range of use cases, but it is not always the right tool. Here is a quick guide to pick the right approach.

Use code execution when you need precise numeric answers, data analysis on structured data, or deterministic transformations. Examples: sums of primes, statistical summaries of datasets, parsing and transforming CSVs, or generating plots and files.
Prefer pure language responses when the task is primarily conceptual, creative, or conversational and does not rely on numerical precision. Examples: brainstorming, creative writing, summarization, or high-level planning.
Combine tools when you need both: let the LLM draft an explanation and generate code for exact computations in the same session. Route code to execution and explanations to a text reply so you get both clarity and correctness.

As a rule of thumb, if you are asking the model for a numeric result that will be used in downstream logic or decision-making, prefer code execution so you can validate the output programmatically.

📈 Real-world use cases I imagine

Beyond small demos, I see a wide range of practical applications for this pattern. Here are a few I thought about while building the demos.

Data explorer for business users. Non-technical users could upload spreadsheets and ask natural language questions. The model generates and executes Python code that pulls the answers from the sheet and presents results with the executed snippet for auditing.
Education and tutoring. Students can ask math problems, see the code that computes the answer, and learn both the algorithm and the programming pattern.
Rapid prototyping. Developers can ask the model to generate data transformation pipelines and immediately run them to inspect outputs. This accelerates exploration and proof-of-concept work.
Reporting automation. Generate scripts that compute summary statistics or build figures, run them on a schedule, and embed both the generated code and results in reports.
Audit trails. In regulated environments generate code for computations that must be auditable. Keep logs of generated code and inputs to satisfy compliance needs.

Each of these use cases gains from the same core strengths: precision, transparency, and speed.

🔍 A deeper look at the technical trade-offs

There are trade-offs you should be aware of when adopting code execution. It is tempting to think of it as a silver bullet. It is not. Here are the most important technical considerations I encountered and how to think about them.

Determinism vs model nondeterminism

The generated code can be different across runs even for the same prompt, because the model sampling process may produce alternative implementations. But once a particular snippet is executed, the runtime behavior is deterministic (within resource limits). Therefore, if reproducibility is important, you need to save both the code and the inputs. If you re-run the same prompt later, you might get a different piece of code unless you fix the model seed or otherwise control decoding.

Performance and latency

Generating code and executing it adds round trips compared to a purely text reply. Execution time depends on the complexity of the generated script and the runtime limits. For quick math or simple data queries the latency is low, but if the model generates expensive computations or large pandas operations you can experience longer waits. Practical apps should enforce timeouts and either stream partial results or notify users when a query will take longer.

Dependency management

When the model asks for libraries, you have to decide which ones to allow. In my demos I relied on standard libraries and pandas. For production systems you may want to maintain a curated environment with preinstalled packages that are common for your domain. Allowing the model to install arbitrary packages at execution time is risky and should be avoided.

Error handling

Generated code will sometimes throw exceptions. Plan for that. The execution environment should capture stack traces and return them to the app, along with the original code, so you can fix bugs or refine prompts. I also added safety checks in the app so that if code raises an exception the model gets the stack trace and can generate a corrected snippet upon request.

✍️ Prompt examples I used and why they work

Below are a few example prompts that I used or would use in similar settings. They are intentionally explicit about the expected output format so parsing is easy and code is short and auditable.

Prime sum prompt

Write runnable Python that computes the sum of the first 60 prime numbers. Print only the final integer and nothing else. Do not include any extra text or explanation.

Why it works: The instruction to print only the final integer avoids verbose language from the model. The model writes a small primality check and a loop, then prints a single number for easy parsing.
CSV aggregation prompt

Load 'transactions.csv' into a pandas DataFrame named df. This file contains columns: product, quantity, price, category. Filter rows where category is 'Fresh Fruit'. Group by product and sum quantity. Print the product name and the integer quantity separated by a comma. Do not print anything else.

Why it works: This prompt defines the variable name, the file name, and the columns so the model can produce minimal, correct code. The strict output format makes parsing trivial.
Data sanity check prompt

Write runnable Python that loads 'transactions.csv', checks for missing values in quantity and product columns, and prints 'OK' if there are no missing values or prints the list of rows with missing values otherwise. Keep the code short and include a short comment at the top describing the steps.

Why it works: The model generates a safety check to validate assumptions which reduces runtime surprises during larger operations.

💡 UI and UX tips for apps that expose code execution

If you expose code execution to end users, design matters. I want users to trust the output and feel safe. Here are UX decisions I made in the CSV Genius prototype and why they help.

Show the generated code by default but collapsed. Users can expand to inspect. This keeps the UI clean while maintaining transparency.
Show execution logs. Capture stdout and any warnings so users can see intermediate information like progress messages or note about dropped rows.
Allow re-execution with edits. Let users edit the generated code in a small editor and re-run it. That supports power users and debugging.
Limit query complexity with guidance. If a user asks a very expensive aggregate over a huge file, show suggestions for sampling or limiting the query first.
Explain permissions. Communicate what the execution environment can and cannot access, especially regarding network calls and file I/O.

These small choices make the difference between a mysterious black box and a trustworthy assistant that produces reproducible results.

🔬 Common failure modes and how I handled them

Not everything ran smoothly. Here are common problems I saw and how I fixed or mitigated them.

Wrong assumptions about the data

Sometimes the model assumed a column existed or that a column was numeric. The code raised an exception. I mitigated this by asking for an initial sanity check snippet that validates column existence and types, returning a clear error message if the assumptions fail.

Infinite loops or heavy computation

On rare occasions the model generated an inefficient algorithm, especially for math tasks when it used naive primality checks for large ranges. I enforced timeouts and resource caps, and I added an instruction to use efficient standard library functions or vectorized pandas operations where appropriate.

Ambiguous output formatting

The model sometimes printed additional explanatory text or debug prints. I enforced strict output formats in the prompt and used post-execution parsing that ignores extraneous content, returning an error to the user when parsing fails so they know to request a corrected snippet.

Security attempts

Although in my small tests I did not see active exploitation, I designed the execution environment to prevent network access and disallowed arbitrary imports. Any attempt by the model to call dangerous operations was blocked by the sandbox and the run returned a clear error indicating the operation was not permitted.

🔗 Integrations with other tools and APIs

The code execution tool is not an island. It plays well with other kinds of tools and services. I experimented conceptually with a few integration patterns I think are promising.

Combining retrieval and execution. Retrieve documents or CSVs via a secure storage API, feed a subset of the data into the execution environment, and let the model run code on that subset. This reduces data transfers and keeps sensitive data in your controlled storage.
Calling specialized services. For heavy numerical tasks you might call a dedicated numeric or statistical service from the sandboxed runtime. That still requires strict whitelisting and monitoring.
Visualization backends. Generate matplotlib or similar plotting code, execute it, and capture the generated image to display in your app. This turns plain text questions into shareable charts.
CI pipelines. Use generated code as part of automated report pipelines. The model drafts the code, tests run, and results are included in scheduled reports. Keep a review step to avoid accidental changes in production pipelines.

🧾 Example of a complete end-to-end scenario

To bring all of this together, here is a narrative example of how an analyst might use code execution in a small app that I built.

I upload monthly_sales.csv to the app. The file contains product, category, quantity, price, and region.
I type: "Which product sold the most units in the North region last month? Show product and units."
The app sends this prompt with code execution enabled and gives the model access to the uploaded CSV as 'monthly_sales.csv'.
The model returns Python that loads the CSV, filters region == 'North', groups by product and sums quantity, then prints "product_name,units".
The execution sandbox runs the code, returns "starfruit,124" as the output and shows the executed code in a collapsed panel.
I expand the code, confirm it filters and groups correctly, and optionally edit it to include an additional filter for 'category == Fresh Fruit'.
I re-run the modified code and get an updated answer. I save the final code as a reproducible artifact for my report.

This flow is compact, auditable, and practical in real analytics workflows.

🔭 Where this tech is heading

Code execution is one of several emerging patterns that combine LLMs with stateful tools. I see a few trends that will shape how developers use this capability.

Better tool orchestration. Models will learn to orchestrate multiple tools in a reliable way: do a quick search, fetch a CSV, run a validation snippet, call a plotting backend, and produce a combined deliverable.
More robust execution sandboxes. Expect richer runtimes with curated package sets, secure connectors to data stores, and built-in monitoring and cost controls.
Verification helpers. Models will generate not only code but also assertions and quick tests that verify assumptions about input data or intermediate results.
Interactive debugging. The model will suggest fixes for failing runs, either automatically or via interactive back-and-forth where it receives the stack trace and returns a corrected snippet.

These advancements will make the model+tool pattern safer, more powerful, and more trustworthy for production use.

🔁 Closing thoughts and quick tips

To summarize my experience:

LLMs are not calculators, but they are very good at writing code.
When you let a model generate code and execute it, you gain accuracy and transparency for numeric tasks and data analysis.
Design prompts carefully to make outputs deterministic and parsable.
Always run generated code inside a sandbox with strict limits and logging.
Save generated code and inputs to ensure reproducibility and to provide an audit trail.

If you are building an app that needs reliable numeric results or wants to make data accessible to non-programmers, consider the model plus code execution pattern. It offers a practical middle ground between pure natural language answers and fully manual coding.

❓ Frequently Asked Questions

What is the code execution tool and why should I use it?

The code execution tool lets a language model generate code and have that code executed in a controlled backend environment. You should use it when you need exact numeric answers, reproducible computations, or transparent data analysis. Instead of relying on the model to guess numbers, you can have the model produce code that runs precise operations in Python or another supported runtime.

Does the model still do reasoning if I use code execution?

Yes. The model still performs reasoning and writes the program that implements the reasoning. The difference is that computationally precise steps are carried out by the runtime, meaning numeric results and data operations are deterministic to the extent the code and environment are deterministic.

What happens if the generated code is malicious or tries to exfiltrate data?

You must run generated code inside a sandbox with strict controls: no outgoing network access, limited filesystem access, resource caps, and curated package lists. The execution environment should monitor and block dangerous operations and log any blocked attempts for audit. Treat generated code as untrusted input and enforce standard runtime security practices.

Can the model install arbitrary packages when it runs code?

In responsible deployments you should not allow arbitrary package installs. Instead, maintain a curated environment with the packages you trust and need. Allowing dynamic installs is risky because packages could contain malice or cause dependency conflicts. If you need a package, preinstall it in the sandboxed runtime after vetting it.

What if the model prints extra text and not a clean machine-parsable result?

Design your prompt to enforce a strict output format. Tell the model to print only the final value or to return JSON. Additionally, your app should include robust parsing logic that can handle slight deviations or return a parsing error to the user requesting the model to correct the format. You can also ask the model to include a short block with only the machine-readable output.

Is execution fast enough for interactive apps?

For small scripts and datasets execution is fast and suitable for interactive apps. If the model generates expensive operations or processes very large datasets, you will see higher latency. Mitigate this with sampling, limits on input sizes, pre-aggregation, or background jobs for heavy queries.

How do I ensure reproducibility of results?

Save the exact code snippet, the prompt used, input files, and the execution metadata such as the runtime version and resource constraints. If you need deterministic generation from the model, also control decoding parameters or explicitly save the generated code. Re-running the saved code in the same environment should produce the same results if the environment and inputs are unchanged.

What kinds of tasks should not use code execution?

Tasks that are primarily creative or conversational and do not require numeric precision are better handled by pure language responses. Also avoid code execution for highly sensitive operations that require direct access to production systems unless you have strict controls and review processes in place.

Can I combine code execution with other tools like search or a calculator?

Yes. Combining tools is a powerful pattern. You can have the model orchestrate search, retrieval, code execution, and other APIs. For example, it could fetch a CSV from secure storage, run Python code to analyze it, then call a visualization service to create a chart. Always control and audit each tool's permissions and data access.

How do I start building with code execution?

Start with small, well-scoped use cases: sums, aggregates, data validation, and simple transforms. Build a safe sandboxed runtime with resource limits. Create clear prompts that specify input names and desired output formats. Log generated code and results and iterate on your prompts and sandbox configuration based on the errors and needs you observe.

📣 Final notes

I am excited about how code execution complements the natural language strengths of models like Gemini. It turns language models into practical partners for computation and data work. By asking models to write code and then executing that code in a secure, controlled environment, you get the best of both worlds: the creativity and reasoning of LLMs and the precision and reproducibility of programmatic computation.

If you build something using this approach, make sure you treat generated code with caution, design for transparency, and log everything you need to reproduce results. With those guardrails in place you can make powerful new tools that bring data analysis and exact computation to a broader audience.

That's the story I wanted to share: a small technical pattern with outsized practical impact. I hope you try it and let me know what you build.

Table of Contents