Barclays EDP Demo — Devin for Data Engineering
Three demos showing how Devin accelerates the data engineering workload that sits between the data-as-a-product vision and production reality.
Data Ingestion + Schema Change
Devin builds a connector from a public data source into a target schema, then handles a schema change automatically — the kind of work that normally takes a data engineer a day or more.
V1 Base Connector — GitHub API → SQLite
Devin writes a Python ingestion script that extracts repo name, description, language, star count, and last updated date from the GitHub API, and loads it into a local SQLite table.
$ python ingest_repos.py
Fetching repositories for 'dbt-labs' from GitHub API...
-> Retrieved 396 repositories.
-> Loaded 396 rows into raw_repos table.
Top 5 repos by stars:
dbt-core 12,450 stars
dbt-utils 1,718 stars
metricflow 1,527 stars
dbt-fusion 667 stars
dbt-codegen 644 stars
Done. Database saved to raw_repos.db
V2 Schema Evolution — Adding forks_count
The upstream source has added a new field. Devin detects the schema change, updates the script, alters the table definition, and re-runs without breaking existing data.
$ python ingest_repos_v2_schema_change.py
Fetching repositories for 'dbt-labs' from GitHub API...
-> Retrieved 396 repositories.
[SCHEMA CHANGE] Adding 'forks_count' column to raw_repos...
[SCHEMA CHANGE] Column added successfully.
-> Loaded 396 rows into raw_repos table (with forks_count).
Top 5 repos by stars (now including forks):
dbt-core 12,450 stars | 2,323 forks
dbt-utils 1,718 stars | 609 forks
metricflow 1,527 stars | 159 forks
dbt-fusion 667 stars | 100 forks
dbt-codegen 644 stars | 131 forks
Done. Database saved to raw_repos.db
The schema migration detected the missing column, added it via ALTER TABLE, and re-ingested without breaking existing data.
Transform SQL into a dbt Model
Devin takes raw SQL transformation logic and converts it into a clean, documented dbt model that maps to a data product contract — the core of migrating from legacy jobs.
Before: Raw SQL (jaffle_shop customers.sql)
with customers as (
select * from {{ ref('stg_customers') }}
),
orders as (
select * from {{ ref('stg_orders') }}
),
payments as (
select * from {{ ref('stg_payments') }}
),
customer_orders as (
select customer_id,
min(order_date) as first_order,
max(order_date) as most_recent_order,
count(order_id) as number_of_orders
from orders
group by customer_id
),
customer_payments as (
select orders.customer_id,
sum(amount) as total_amount
from payments
left join orders on payments.order_id = orders.order_id
group by orders.customer_id
),
final as (
select customers.customer_id, customers.first_name, customers.last_name,
customer_orders.first_order, customer_orders.most_recent_order,
customer_orders.number_of_orders,
customer_payments.total_amount as customer_lifetime_value
from customers
left join customer_orders on customers.customer_id = customer_orders.customer_id
left join customer_payments on customers.customer_id = customer_payments.customer_id
)
select * from final
After: Clean dbt Model with Derived Fields
Devin rewrites the model with clear structure, self-documenting comments, and adds the
customer_status field required by the Customer Profiles data product contract.
{{ config(materialized = 'table', tags = ['customer_profiles', 'data_product']) }}
-- Step 1: Pull base customer attributes from the staging layer.
with customers as (
select customer_id, first_name, last_name
from {{ ref('stg_customers') }}
),
-- Step 2: Aggregate order-level metrics per customer.
customer_orders as (
select customer_id,
min(order_date) as first_order_date,
max(order_date) as most_recent_order_date,
count(order_id) as number_of_orders
from {{ ref('stg_orders') }}
group by customer_id
),
-- Step 3: Aggregate payment totals per customer.
customer_payments as (
select orders.customer_id,
sum(payments.amount) as total_amount
from {{ ref('stg_payments') }} as payments
left join {{ ref('stg_orders') }} as orders
on payments.order_id = orders.order_id
group by orders.customer_id
),
-- Step 4: Join all CTEs and compute derived fields.
final as (
select
customers.customer_id,
customers.first_name,
customers.last_name,
customer_orders.first_order_date,
customer_orders.most_recent_order_date,
customer_orders.number_of_orders,
coalesce(customer_payments.total_amount, 0) as customer_lifetime_value,
-- Derived: customer_status (required by Customer Profiles contract)
case
when customer_orders.customer_id is null then 'prospect'
when customer_orders.number_of_orders = 1 then 'new'
when most_recent_order_date >= dateadd('day', -90, current_date)
then 'active'
else 'lapsed'
end as customer_status
from customers
left join customer_orders on customers.customer_id = customer_orders.customer_id
left join customer_payments on customers.customer_id = customer_payments.customer_id
)
select * from final
Generated schema.yml — Catalogue-Ready Documentation
Everything needed to publish this to a data catalogue — column names, descriptions, and data quality tests — generated automatically as a by-product of the build.
| Column | Description | Tests |
|---|---|---|
customer_id | Primary key — unique identifier | unique, not_null |
first_name | Customer's first name | not_null |
last_name | Customer's last name | not_null |
first_order_date | Date of earliest order | — |
most_recent_order_date | Date of most recent order | — |
number_of_orders | Total orders placed | — |
customer_lifetime_value | Total payment amount (base currency) | not_null |
customer_status |
Derived: prospect / new / active / lapsed | not_null, accepted_values |
7 data quality tests total across all columns.
Pipeline Triage
Devin investigates a broken dbt model, identifies the root cause, patches the test, and explains what it did — the kind of first-response triage that currently pulls engineers away from product work.
FIX 1 Autonomous Fix — Column Typo
CI Error: column 'custmer_id' does not exist
-- BROKEN:
SELECT customer_id, SUM(amount) AS total_revenue, COUNT(order_id) AS order_count
FROM {{ ref('orders') }}
GROUP BY custmer_id ← typo: missing 'o'
-- FIXED (by Devin):
SELECT customer_id, SUM(amount) AS total_revenue, COUNT(order_id) AS order_count
FROM {{ ref('orders') }}
GROUP BY customer_id ← corrected
Resolution: Autonomous fix — no business decision required. Devin spots the typo, explains why it fails, and produces the corrected model.
FIX 2 Escalation — Guest Checkout Handling
CI Error: not_null test failed for customer_id in orders model
Root Cause: The source table recently added rows where
customer_id can be NULL for guest checkouts.
| Option | Approach | Pros | Cons |
|---|---|---|---|
| A | Filter out NULLs | Simple, preserves test | Loses guest order data |
| B (Recommended) | COALESCE to sentinel (-1) | Preserves all data, keeps test | Magic number downstream |
| C | Separate guest model | Cleanest design | Requires downstream refactoring |
-- RECOMMENDED FIX (Option B):
SELECT
order_id,
coalesce(customer_id, -1) AS customer_id, -- guest sentinel
order_date,
status,
amount,
case when customer_id is null then true
else false end as is_guest_checkout -- downstream flag
FROM {{ ref('stg_orders') }}