Devin for Data Engineering

Enter the 4-digit access code to view the demo

Barclays EDP Demo — Devin for Data Engineering

Three demos showing how Devin accelerates the data engineering workload that sits between the data-as-a-product vision and production reality.

Demo 1 — Ingestion
Demo 2 — dbt Transform
Demo 3 — Pipeline Triage

Data Ingestion + Schema Change

Devin builds a connector from a public data source into a target schema, then handles a schema change automatically — the kind of work that normally takes a data engineer a day or more.

V1 Base Connector — GitHub API → SQLite

Devin writes a Python ingestion script that extracts repo name, description, language, star count, and last updated date from the GitHub API, and loads it into a local SQLite table.

$ python ingest_repos.py

Fetching repositories for 'dbt-labs' from GitHub API...
  -> Retrieved 396 repositories.
  -> Loaded 396 rows into raw_repos table.

Top 5 repos by stars:
  dbt-core                       12,450 stars
  dbt-utils                       1,718 stars
  metricflow                      1,527 stars
  dbt-fusion                        667 stars
  dbt-codegen                       644 stars

Done. Database saved to raw_repos.db

V2 Schema Evolution — Adding forks_count

The upstream source has added a new field. Devin detects the schema change, updates the script, alters the table definition, and re-runs without breaking existing data.

$ python ingest_repos_v2_schema_change.py

Fetching repositories for 'dbt-labs' from GitHub API...
  -> Retrieved 396 repositories.

  [SCHEMA CHANGE] Adding 'forks_count' column to raw_repos...
  [SCHEMA CHANGE] Column added successfully.

  -> Loaded 396 rows into raw_repos table (with forks_count).

Top 5 repos by stars (now including forks):
  dbt-core                       12,450 stars  |  2,323 forks
  dbt-utils                       1,718 stars  |    609 forks
  metricflow                      1,527 stars  |    159 forks
  dbt-fusion                        667 stars  |    100 forks
  dbt-codegen                       644 stars  |    131 forks

Done. Database saved to raw_repos.db

The schema migration detected the missing column, added it via ALTER TABLE, and re-ingested without breaking existing data.

Transform SQL into a dbt Model

Devin takes raw SQL transformation logic and converts it into a clean, documented dbt model that maps to a data product contract — the core of migrating from legacy jobs.

Before: Raw SQL (jaffle_shop customers.sql)

with customers as (
    select * from {{ ref('stg_customers') }}
),
orders as (
    select * from {{ ref('stg_orders') }}
),
payments as (
    select * from {{ ref('stg_payments') }}
),
customer_orders as (
    select customer_id,
        min(order_date) as first_order,
        max(order_date) as most_recent_order,
        count(order_id) as number_of_orders
    from orders
    group by customer_id
),
customer_payments as (
    select orders.customer_id,
        sum(amount) as total_amount
    from payments
    left join orders on payments.order_id = orders.order_id
    group by orders.customer_id
),
final as (
    select customers.customer_id, customers.first_name, customers.last_name,
        customer_orders.first_order, customer_orders.most_recent_order,
        customer_orders.number_of_orders,
        customer_payments.total_amount as customer_lifetime_value
    from customers
    left join customer_orders on customers.customer_id = customer_orders.customer_id
    left join customer_payments on customers.customer_id = customer_payments.customer_id
)
select * from final

After: Clean dbt Model with Derived Fields

Devin rewrites the model with clear structure, self-documenting comments, and adds the customer_status field required by the Customer Profiles data product contract.

{{ config(materialized = 'table', tags = ['customer_profiles', 'data_product']) }}

-- Step 1: Pull base customer attributes from the staging layer.
with customers as (
    select customer_id, first_name, last_name
    from {{ ref('stg_customers') }}
),

-- Step 2: Aggregate order-level metrics per customer.
customer_orders as (
    select customer_id,
        min(order_date) as first_order_date,
        max(order_date) as most_recent_order_date,
        count(order_id) as number_of_orders
    from {{ ref('stg_orders') }}
    group by customer_id
),

-- Step 3: Aggregate payment totals per customer.
customer_payments as (
    select orders.customer_id,
        sum(payments.amount) as total_amount
    from {{ ref('stg_payments') }} as payments
    left join {{ ref('stg_orders') }} as orders
        on payments.order_id = orders.order_id
    group by orders.customer_id
),

-- Step 4: Join all CTEs and compute derived fields.
final as (
    select
        customers.customer_id,
        customers.first_name,
        customers.last_name,
        customer_orders.first_order_date,
        customer_orders.most_recent_order_date,
        customer_orders.number_of_orders,
        coalesce(customer_payments.total_amount, 0) as customer_lifetime_value,

        -- Derived: customer_status (required by Customer Profiles contract)
        case
            when customer_orders.customer_id is null then 'prospect'
            when customer_orders.number_of_orders = 1  then 'new'
            when most_recent_order_date >= dateadd('day', -90, current_date)
                then 'active'
            else 'lapsed'
        end as customer_status

    from customers
    left join customer_orders on customers.customer_id = customer_orders.customer_id
    left join customer_payments on customers.customer_id = customer_payments.customer_id
)
select * from final

Generated schema.yml — Catalogue-Ready Documentation

Everything needed to publish this to a data catalogue — column names, descriptions, and data quality tests — generated automatically as a by-product of the build.

ColumnDescriptionTests
customer_idPrimary key — unique identifierunique, not_null
first_nameCustomer's first namenot_null
last_nameCustomer's last namenot_null
first_order_dateDate of earliest order—
most_recent_order_dateDate of most recent order—
number_of_ordersTotal orders placed—
customer_lifetime_valueTotal payment amount (base currency)not_null
customer_status Derived: prospect / new / active / lapsed not_null, accepted_values

7 data quality tests total across all columns.

Pipeline Triage

Devin investigates a broken dbt model, identifies the root cause, patches the test, and explains what it did — the kind of first-response triage that currently pulls engineers away from product work.

FIX 1 Autonomous Fix — Column Typo

CI Error: column 'custmer_id' does not exist

-- BROKEN:
SELECT customer_id, SUM(amount) AS total_revenue, COUNT(order_id) AS order_count
FROM {{ ref('orders') }}
GROUP BY custmer_id    ← typo: missing 'o'

-- FIXED (by Devin):
SELECT customer_id, SUM(amount) AS total_revenue, COUNT(order_id) AS order_count
FROM {{ ref('orders') }}
GROUP BY customer_id   ← corrected

Resolution: Autonomous fix — no business decision required. Devin spots the typo, explains why it fails, and produces the corrected model.


FIX 2 Escalation — Guest Checkout Handling

CI Error: not_null test failed for customer_id in orders model

Root Cause: The source table recently added rows where customer_id can be NULL for guest checkouts.

OptionApproachProsCons
AFilter out NULLsSimple, preserves testLoses guest order data
B (Recommended) COALESCE to sentinel (-1)Preserves all data, keeps testMagic number downstream
CSeparate guest modelCleanest designRequires downstream refactoring
-- RECOMMENDED FIX (Option B):
SELECT
    order_id,
    coalesce(customer_id, -1) AS customer_id,   -- guest sentinel
    order_date,
    status,
    amount,
    case when customer_id is null then true
         else false end as is_guest_checkout      -- downstream flag
FROM {{ ref('stg_orders') }}
Escalation Boundary: Devin recommends Option B but asks for engineer confirmation before applying. The choice of sentinel value (-1 vs. a dedicated 'guest' dimension key) is a business decision. This is what makes Devin safe to run autonomously — it knows when a fix requires a human decision and stops to ask.