Analytics Engineers and Machine Learning: How They Work Together

How do they work together to ensure model reliability?

Analytics Engineers and Machine Learning teams build a shared data contract where the Analytics Engineer provides clean, versioned, and documented data assets while the Machine Learning team consumes these assets for model training and inference. In our experience, they work together most effectively when they treat the data warehouse as a shared feature store rather than a dumping ground for raw tables.

This collaboration allows the Machine Learning (ML) team to focus on model architecture and hyperparameter tuning while the Analytics Engineer (AE) handles the upstream data quality, transformation logic, and lineage. By establishing clear ownership at the point where data leaves the dbt (Data Build Tool) model and enters the ML pipeline, teams can avoid the common "garbage in, garbage out" problem that plagues many AI initiatives.

The Role of Analytics Engineering in the ML Lifecycle

In the traditional data hierarchy, the Analytics Engineer sits between the Data Engineer and the Data Analyst. However, in an AI-forward organization, the Analytics Engineer also serves as the primary provider for the Data Scientist. This analytics engineer ml collaboration ensures that the data used for predictive modeling is the exact same data used for executive reporting.

When these roles are siloed, the Data Science team often ends up writing custom SQL queries to pull data from raw staging tables. This creates a "logic gap" where the Revenue numbers in the CEO dashboard do not match the Revenue numbers used to train a churn prediction model. Our team advocates for a "Warehouse-First" approach where all logic is centralized in the analytics layer before it ever reaches a model.

If you are currently struggling with conflicting metrics across your stack, our AI Stack Audit provides a scored assessment of your data foundation to identify where these gaps exist.

Bridging the Gap Between Analytics Engineering and Data Science

The bridge between analytics engineering and data science is built on the concept of the "feature table." A feature table is a curated dbt model that gathers all relevant attributes for a specific entity, such as a customer or a product, into a single wide table.

Instead of the Data Scientist performing complex joins in a Jupyter notebook, the Analytics Engineer builds these joins into a production-grade dbt model. This approach offers three distinct advantages:

1. Reusability: The same "customer_features" table can be used for five different ML models.
2. Version Control: Any change to the feature logic is captured in Git, allowing for easy audits and rollbacks.
3. Consistency: The logic for "active user" is defined once in SQL and used everywhere.

For example, an Analytics Engineer might build a fct_customer_features model that looks like this:

-- dbt model: fct_customer_features
WITH orders AS (
    SELECT * FROM {{ ref('stg_orders') }}
),
customer_metrics AS (
    SELECT
        customer_id,
        COUNT(order_id) AS total_orders,
        SUM(revenue) AS total_lifetime_revenue,
        MAX(order_date) AS last_order_date,
        DATEDIFF('day', MAX(order_date), CURRENT_DATE()) AS days_since_last_order
    FROM orders
    GROUP BY 1
)
SELECT
    c.customer_id,
    c.customer_segment,
    m.total_orders,
    m.total_lifetime_revenue,
    m.days_since_last_order,
    CASE WHEN m.days_since_last_order <= 30 THEN 1 ELSE 0 END AS is_active_30d
FROM {{ ref('dim_customers') }} c
LEFT JOIN customer_metrics m ON c.customer_id = m.customer_id

This model provides a clean interface for the Data Scientist to start their work immediately. This is a core component of the Data Foundation track we teach, focusing on how to build dbt models that serve both BI (Business Intelligence) and ML (Machine Learning).

Technical Workflows: dbt and Feature Engineering

While the Analytics Engineer builds the foundation, the ML Engineer or Data Scientist often needs more granular control over specific feature transformations like one-hot encoding or scaling. In a mature analytics engineer machine learning role, the AE manages the "upstream" features (the raw facts) while the ML team manages the "downstream" features (the transformations specific to the model).

We recommend using dbt for the upstream work because it handles dependency management and documentation natively. If the stg_orders table fails a data quality test, the downstream ML training pipeline should be alerted or paused. This prevents the model from training on corrupted data, which is a major cause of silent failures in production AI systems.

Managing Data Contracts and Documentation

A data contract is a formal agreement between the AE and the ML team regarding the schema, quality, and freshness of a table. If the Analytics Engineer needs to rename a column in the warehouse, they must check if that column is a feature for a production model.

Without this coordination, simple warehouse cleanups can break AI products. We solve this by using dbt's meta-tags and exposures. By tagging models that are used by ML pipelines, the AE can see the downstream impact of their changes in the dbt lineage graph.

The Importance of Data Quality Tests

Data quality is the most significant bottleneck in AI performance. We typically implement four layers of testing when building these collaborative systems:

1. Schema Tests: Ensuring IDs are not null and values are within expected ranges.
2. Relationship Tests: Confirming that every order belongs to a valid customer.
3. Distribution Tests: Monitoring if the average value of a feature has shifted significantly (indicating potential drift).
4. Freshness Tests: Verifying that the data was updated within the last 24 hours.

When we deploy these systems for our clients, we often find that 80 percent of model performance issues are actually data quality issues in disguise.

Common Pitfalls in Collaboration

Even with the best tools, teams can fail if their workflows are not aligned. Here are the most common mistakes we see:

1. The Jupyter Notebook Trap A Data Scientist develops a model in a notebook using a local CSV export. When it comes time to move to production, the Analytics Engineer has to "reverse engineer" the logic back into SQL. This process is slow and prone to errors. Instead, the Data Scientist should be querying the production warehouse from day one.

2. Over-Engineering the Feature Store Many startups try to implement a standalone feature store before they have even mastered basic dbt modeling. In our experience, a well-structured BigQuery or Snowflake instance acting as a "silver" or "gold" layer is sufficient for 90 percent of mid-market use cases.

3. Ignoring Data Lineage When a model starts producing strange results, the first question is always: "Where did this data come from?" If the AE has not documented the lineage from the source API to the final feature table, debugging becomes a multi-day ordeal involving multiple team members.

Frequently Asked Questions About Analytics Engineering and ML

What is the difference between an Analytics Engineer and a Machine Learning Engineer?

An Analytics Engineer focuses on the reliability, quality, and accessibility of data within the warehouse using SQL and dbt. A Machine Learning Engineer focuses on the infrastructure required to train, deploy, and monitor predictive models using Python and specialized ML platforms. They work together by sharing a common data layer where the AE provides the input and the MLE provides the output.

Should the Analytics Engineer or the Data Scientist write the SQL for features?

In a high-performing team, the Analytics Engineer writes the core SQL logic to ensure it follows engineering best practices like DRY (Don't Repeat Yourself) and modularity. The Data Scientist provides the requirements for what features are needed. This prevents the creation of "spaghetti SQL" that is difficult to maintain in a production environment.

Can dbt be used for Machine Learning pipelines?

dbt is excellent for the preprocessing and feature engineering stages of an ML pipeline. However, dbt is not designed for model training or hosting. We recommend using dbt to prepare the feature tables and then using a tool like Vertex AI, SageMaker, or a custom Python script to handle the actual machine learning tasks.

How do you handle data drift in this collaboration?

Data drift occurs when the statistical properties of the input data change over time. The Analytics Engineer can help detect this by implementing distribution tests in dbt (using packages like dbt_expectations). If the mean value of a "discount_amount" feature changes significantly, the AE can alert the ML team before the model's accuracy drops.

When should a company hire an Analytics Engineer for their ML team?

You should hire an Analytics Engineer when your Data Scientists are spending more than 50 percent of their time cleaning data and writing repetitive SQL queries. If your ML models are failing because of upstream data changes or if your warehouse has no clear documentation, an Analytics Engineer is the right hire to unblock your AI roadmap.

Ready to strengthen your data foundation?

The relationship between analytics engineering and machine learning is the bedrock of any successful AI strategy. If your team is struggling to move models into production because of messy data or fragile pipelines, we can help you bridge that gap.

Our team at MLDeep Systems specializes in building the technical infrastructure that makes AI possible. We offer an AI Stack Audit for teams that need a clear roadmap of their current gaps, as well as a Learn AI Bootcamp for data professionals who want to master the tools required for production AI.

Whether you need to clean up your dbt models or deploy your first production AI agent, we provide the practitioner-led guidance to get you there. Book a free consultation to talk through your current data architecture and identify the fastest path to reliable, automated insights.