Unit vs Integration vs End-to-End Tests Explained

A function can have 100% unit test coverage and the page that calls it can still return a 500 in production. That is not a contradiction. Unit tests, integration tests, and end-to-end tests check different things, and a suite that’s heavy on one and light on another has a specific, predictable hole in it.

What unit, integration, and end-to-end tests each check

Take a sign-up feature: a function validates the input, writes a row to a database, and sends a welcome email. Three tests can cover it, and each one answers a different question.

A unit test tells you the email-format check rejects what it should. An integration test tells you the row actually lands in the database with the columns you think it has. An end-to-end test tells you a client can call the endpoint and get a 201 back. None of them can stand in for the others.

Testing one Flask feature at all three levels

Here’s the feature: a sign_up function that validates an email, writes a user, and sends a welcome message.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

# app.py
import os
import smtplib
import sqlite3

from flask import Flask, jsonify, request


def get_db():
    db_path = os.environ.get("DB_PATH", "app.db")
    return sqlite3.connect(db_path)


def get_smtp_client():
    host = os.environ.get("SMTP_HOST", "localhost")
    port = int(os.environ.get("SMTP_PORT", "25"))
    return smtplib.SMTP(host, port)


def init_db(db):
    db.execute("CREATE TABLE IF NOT EXISTS users (email TEXT PRIMARY KEY, name TEXT)")
    db.commit()


def create_user(db, email, name):
    db.execute("INSERT INTO users (email, name) VALUES (?, ?)", (email, name))
    db.commit()


def send_welcome_email(smtp_client, to_address, name):
    smtp_client.sendmail(
        "[email protected]",
        to_address,
        f"Subject: Welcome\n\nHi {name}, thanks for signing up.",
    )


def sign_up(db, smtp_client, email, name):
    if not email or "@" not in email:
        raise ValueError("invalid email")
    create_user(db, email, name)
    send_welcome_email(smtp_client, email, name)
    return {"email": email, "name": name, "status": "created"}


app = Flask(__name__)


@app.post("/signup")
def signup_route():
    data = request.get_json()
    db = get_db()
    init_db(db)
    smtp_client = get_smtp_client()
    result = sign_up(db, smtp_client, data["email"], data["name"])
    smtp_client.quit()
    return jsonify(result), 201


@app.get("/users/<email>")
def get_user_route(email):
    db = get_db()
    row = db.execute("SELECT email, name FROM users WHERE email = ?", (email,)).fetchone()
    if row is None:
        return jsonify({"error": "not found"}), 404
    return jsonify({"email": row[0], "name": row[1]})

sign_up takes db and smtp_client as arguments instead of reaching for a global connection. That one choice makes all three tests below possible: each passes something different for those two parameters. get_db() and get_smtp_client() read their target from an environment variable on every call instead of hardcoding it once, which is what lets the end-to-end test below point the running app at a throwaway database and a local test mail server without touching app.py.

Unit test: swap out both dependencies

A unit test isolates sign_up completely. Neither the database nor the SMTP client is real; both are Mock objects from Python’s standard library.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# test_unit.py
from unittest.mock import Mock

import pytest

from app import sign_up


def test_sign_up_rejects_invalid_email():
    db, smtp_client = Mock(), Mock()
    with pytest.raises(ValueError):
        sign_up(db, smtp_client, "not-an-email", "Ada")
    db.execute.assert_not_called()


def test_sign_up_sends_welcome_email():
    db, smtp_client = Mock(), Mock()
    sign_up(db, smtp_client, "[email protected]", "Ada")
    smtp_client.sendmail.assert_called_once()
    to_address = smtp_client.sendmail.call_args.args[1]
    assert to_address == "[email protected]"

Two kinds of test double are at work here, and the names matter because they set different expectations. db in the first test is close to a dummy: it’s passed in because the signature needs something, and the test only cares that it’s never touched. smtp_client in the second test is a mock in the strict sense. The test defines the expectation (sendmail gets called with this address) and fails if reality doesn’t match. That’s behavior verification: you check that a call happened, not what came back from it.

These tests run in milliseconds and never touch disk or a socket. They also can’t tell you whether INSERT INTO users matches the real table’s columns. db never executes anything real, so a typo in the SQL would pass every unit test you write.

Integration test: a real database, a fake mail server

An integration test closes that gap one dependency at a time. Here the database is real, an in-memory SQLite connection with the actual schema, while the mail server is still swapped out. This time it’s a fake: a small working implementation instead of a scripted stand-in.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# test_integration.py
import sqlite3

from app import init_db, sign_up


class FakeSMTP:
    def __init__(self):
        self.sent = []

    def sendmail(self, from_addr, to_addr, message):
        self.sent.append((to_addr, message))


def test_sign_up_persists_user_and_records_email():
    db = sqlite3.connect(":memory:")
    init_db(db)
    smtp_client = FakeSMTP()

    sign_up(db, smtp_client, "[email protected]", "Grace")

    row = db.execute(
        "SELECT email, name FROM users WHERE email = ?", ("[email protected]",)
    ).fetchone()
    assert row == ("[email protected]", "Grace")
    assert smtp_client.sent[0][0] == "[email protected]"

FakeSMTP has no expectations scripted into it the way smtp_client.sendmail.assert_called_once() did. It keeps a real, if tiny, list of what was sent, and the test reads that list afterward. That’s the difference between a mock and a fake: a mock is told what to expect before the code runs; a fake behaves like a smaller version of the real thing and you inspect it after.

This test catches a wrong column name or a missing table. It won’t catch a bug in Flask’s request parsing or a wrong URL in the route decorator, because nothing here goes through HTTP.

End-to-end test: a real request against a running process

An end-to-end test drops both substitutions. It starts the actual Flask app as a separate process, sends it a real HTTP request over a real socket, and checks the result the way an outside client would: by calling the API back, not by reaching into the database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

# test_e2e.py
import multiprocessing
import os
import tempfile
import time

import requests
from aiosmtpd.controller import Controller

from app import app


class AcceptAllHandler:
    async def handle_DATA(self, server, session, envelope):
        return "250 Message accepted for delivery"


def run_server():
    app.run(port=5055)


def test_signup_end_to_end():
    db_fd, db_path = tempfile.mkstemp(suffix=".db")
    os.close(db_fd)
    os.environ["DB_PATH"] = db_path
    os.environ["SMTP_HOST"] = "127.0.0.1"
    os.environ["SMTP_PORT"] = "1025"

    mail_server = Controller(AcceptAllHandler(), hostname="127.0.0.1", port=1025)
    mail_server.start()

    server = multiprocessing.Process(target=run_server)
    server.start()
    time.sleep(0.5)  # give Flask's dev server a moment to bind the port

    try:
        signup = requests.post(
            "http://127.0.0.1:5055/signup",
            json={"email": "[email protected]", "name": "Amelia"},
        )
        assert signup.status_code == 201

        lookup = requests.get("http://127.0.0.1:5055/users/[email protected]")
        assert lookup.json() == {"email": "[email protected]", "name": "Amelia"}
    finally:
        server.terminate()
        mail_server.stop()
        os.remove(db_path)

aiosmtpd stands in for the mail-catcher you’d run in a staging environment: something that actually speaks SMTP on a socket, so the app’s real smtplib.SMTP(...) call has somewhere to connect. The test also gives itself a throwaway database file instead of reusing app.db — running this test twice against a shared file would fail the second time on the same PRIMARY KEY that [email protected] already occupies, which is the exact kind of leftover-state flakiness this article gets to further down. Nothing in this test knows that sign_up or create_user exist. It only knows the HTTP contract. Rename a route or change the JSON shape and this is the test that fails, while the unit and integration tests above keep passing.

Install the dependencies with pip install flask requests aiosmtpd pytest and run all three with pytest. When something breaks, which file fails tells you which class of bug you have. That is the reason to write more than one level; coverage percentage isn’t.

Testing pyramid vs. testing trophy

The classic shape for a test suite is Mike Cohn’s testing pyramid: a wide base of fast unit tests, a smaller layer of integration tests, and a thin cap of slow, expensive end-to-end tests. The point is the ratio, not the count. Most of your tests should be the cheap kind.

Kent C. Dodds’ testing trophy disagrees about where the weight goes. His argument: unit tests in isolation give you confidence that’s cheap to earn but doesn’t reflect how the software is used, so a suite made mostly of unit tests can miss real bugs at the seams. The trophy puts static analysis (linting, type checking) at the base, a unit layer above it, integration tests as the widest part, and a thin end-to-end layer on top. test_sign_up_persists_user_and_records_email above is the kind of test the trophy wants more of.

Don’t follow either shape literally. They agree on the ordering: end-to-end tests are the most expensive and should be the fewest, unit tests are the cheapest. Where the bulk of your effort goes depends on your bug history. If most bugs are components misbehaving on their own, lean toward the pyramid. If most are components disagreeing about how they talk to each other, lean toward the trophy.

Why end-to-end tests go flaky

A suite with a growing pile of end-to-end tests eventually earns a reputation: rerun the failing one and it passes. That’s flakiness, and it comes from the same property that makes end-to-end tests valuable. They touch real infrastructure.

The usual causes are a fixed sleep() that’s long enough on your laptop and too short on a loaded CI runner, test data left behind by a previous run colliding with the next one, a network call to something outside your control, and shared state between tests that only shows up when they run in a different order or in parallel. The time.sleep(0.5) in the test above is the first kind; the temp file the same test creates for DB_PATH is a direct fix for the second. None of these are bugs in the feature. They’re bugs in how the test deals with real infrastructure, a problem unit tests never have.

Retrying until green is not the fix; it hides the next real failure in the noise. Keep the end-to-end layer thin by covering the critical paths, not every branch. Give each test its own data instead of shared fixtures. Replace fixed waits with polling for the condition you actually need, such as the port accepting connections. CI time is the other cost: if you’ve already sped up the pipeline by caching dependencies in GitHub Actions, an end-to-end layer that needs three reruns to go green gives most of that time back.

Which kind of test to write next

If a bug just escaped to production, ask which layer would have caught it before you add any test. A bad calculation or an unhandled edge case in one function calls for a unit test. Two pieces of your own code disagreeing about a shape or a contract, like the API returning a field the frontend doesn’t expect or a query that assumes a column that got renamed, calls for an integration test. A feature that passes every test but fails for a real user is usually missing end-to-end coverage of the path that user took.

Write the unit test first. It’s the cheapest signal and the fastest to tell you when you’ve broken the logic. Add an integration test when two of your own pieces need to prove they agree. Save end-to-end tests for the handful of flows where “it works” only means something from the outside: sign-up, login, checkout, password reset.

Keep that split and you get a suite whose failures mean something. That matters beyond CI. Git bisect with git bisect run needs a test command that fails only when the code is actually broken, and a flaky end-to-end test will send it to the wrong commit.

Related articles

• How to Use Git Bisect to Find a Broken Commit
• How to Cache Dependencies in GitHub Actions