Foundations

What this stage actually is

Foundations is not a stage you pass through quickly to get to the interesting parts. It is the stage that determines whether everything after it is built on solid ground or on sand. Spend the time here. The practitioners who move fastest in Stages 1 and 2 are almost always the ones who did Foundations properly — not the ones who skimmed it.

What you are building is not expertise in any one tool. You are building the mental model of how computation works: how data is represented, how programs execute, how files relate to programs, how a database stores and retrieves information. With that model in place, every new library or language you encounter becomes an instance of something you already understand — not a new thing to memorise.

What this stage is not. It is not a computer science degree. You do not need to implement a red-black tree or prove correctness theorems. The bar is practical competence: you can write a small program from scratch, navigate the terminal without dread, manage a project in Git, query a relational database, and consume an API. That is the floor. Everything above it is bonus.

Which path to take. If you have never written a line of code, CS50x is the right entry point — it is the best foundations course on the internet, and it will give you the full mental model in one rigorous run. If you already know another language and want fast Python utility, start with Automate the Boring Stuff. If you want the language reference rather than a course, the Python official tutorial is the authoritative source. All three are free.

Do not let Foundations become a comfortable refuge. It is possible to spend six months here refining your Python style while never touching real data. The exit criterion is not “comfortable with everything.” It is: can you build the project below? If yes, move on.

Core concepts to master

Depth tags: Competent = can do independently, Production = reliable under pressure, Expert = can teach it and debug edge cases.

Unfamiliar with a term? The glossary defines every concept used across the map.

• How computers represent dataCompetent

  Every bug involving integers overflowing, floats comparing unequally, or strings not matching traces back to not knowing this. Understanding bits, bytes, and types turns mysterious errors into predictable ones.

• Variables, types, and control flowCompetent

  The alphabet of every program. You cannot write anything without variables and branching logic; fluency here is what separates 'I read some Python' from 'I can actually code.'

• Functions and modularityCompetent

  The single practice that separates a 50-line script that works once from a 500-line codebase you can maintain. If you cannot break a problem into functions, you cannot build anything real.

• Data structures — lists, dicts, sets, tuplesCompetent

  Every data manipulation you will ever do — from a pandas groupby to a graph traversal — is built from these four. Know when each is appropriate and what operations are fast on each.

• Files and formats — CSV, JSON, plain textCompetent

  Data lives in files. Reading a CSV without pandas, writing clean JSON, parsing a log file: these are the most common tasks in the field and are learned in an afternoon — but you still have to learn them.

• The command line and shellCompetent

  Every server, every pipeline, every deployment lives in a terminal. If you cannot navigate the shell, you cannot work in any production environment — and you will be slower in every environment.

• Version control with GitCompetent

  Code without Git is a Jenga tower — one bad edit and you cannot go back. Git is not a nice-to-have; it is the professional minimum, and the branching model is what makes collaboration possible.

• How the internet works — HTTP, APIs, JSONCompetent

  Every data source you will consume — public datasets, ML APIs, production databases — speaks HTTP. Understanding a request-response cycle lets you pull data from anywhere.

• SQL fundamentals — SELECT, WHERE, GROUP BY, JOINCompetent

  The majority of enterprise data lives in relational databases. SQL is how you access it. The analyst who cannot write a JOIN is locked out of most real-world data before they start.

• Algorithmic thinking and Big-O basicsCompetent

  Understanding why a nested loop over a million rows is slow, and how to fix it, is what separates code that works on a sample from code that works on production data.

Mathematics required

Minimum — what you must know to follow the rest of the map

Arithmetic and algebra fluency. Variables as unknowns, solving for x, reading an equation. The ability to follow a mathematical argument that uses basic algebraic manipulation. This sounds trivially low, but many people discover they have gaps when they first see ML notation — address them now, before the math gets harder.

Functions and graphs. What it means for y to be a function of x. How to read a graph. Logarithms and exponentials — why a log scale compresses large ranges, what happens when you take the log of a probability. These appear constantly in ML (log loss, the log-sum-exp trick, exponential growth curves) and being comfortable with them makes Stage 2 materially less confusing.

Basic set theory and logic. Sets, membership (∈), unions (∪), intersections (∩), and the difference between AND and OR. The vocabulary of logic (if P then Q; the contrapositive; a counterexample). You will see this notation from Stage 1 onward.

Summation notation. What Σ means, how to read an index, and how to evaluate a simple sum. This is the notation of averages, totals, and most statistical formulas. One afternoon of practice is all it takes.

Descriptive statistics vocabulary. Mean, median, mode, variance, standard deviation, and why they differ. The ability to compute these by hand (not just in pandas) so you know what you are asking the computer to do.

Research-grade seed — start building this habit now

You do not need these for Foundations. But starting to build mathematical literacy here — even slowly — compounds enormously by Stage 2 and beyond. The investment is small; the payoff is large.

What a proof is.Read a proof by induction. Read a proof by contradiction. You do not need to write proofs at this stage, but understanding what “prove” means — that a statement holds for all cases, not just the cases you checked — changes how you think about whether your code is correct. This is the foundational epistemological move in mathematics.

Mathematical notation literacy.The symbols ∀ (“for all”), ∃ (“there exists”), ⊆ (“is a subset of”), ⟹ (“implies”), and iff (“if and only if”) are the vocabulary of every textbook from Stage 2 onward. You do not need fluency now — but exposure. When you encounter them in a paper or textbook, do not skip; decode.

→ Tier 1 of the mathematics curriculum covers these topics in full with worked examples — algebra, notation, summation, and proof technique.

Tools and engineering skills

Python:the language of data science, ML engineering, and AI. Learn it properly — not just “I know how to run a script” but truly comfortable with functions, classes, list comprehensions, error handling, modules, and the standard library. Fluency means you spend your mental energy on the problem, not on the syntax.

The terminal: every server, pipeline, and deployment runs in a shell. Know how to navigate directories, read and write files, chain commands with pipes, write simple shell scripts, and understand environment variables. The Missing Semester course covers exactly this gap — the things nobody teaches.

Git and GitHub: not just git commit, but the mental model — commits as a directed acyclic graph, branches as pointers, merges and rebases. Know what a PR is and why it exists. Every project you build from here on lives in a Git repository with a clear commit history. Starting this habit now costs nothing; not starting it costs credibility.

Jupyter Notebooks: the interactive environment where most data work begins. Know how to structure a notebook so it runs reproducibly top-to-bottom, not as a pile of out-of-order cells. A notebook that only works if you ran the cells in a specific secret order is not a notebook — it is a liability.

SQL: the language of relational databases, and how most production data is stored. You do not need a full database administration course — but you need to write SELECT statements with JOINs, GROUP BY, and subqueries without looking everything up. SQLite is available everywhere and sufficient for learning.

VS Code: the practical standard for Python development. Set up a linter (Pylint or Flake8), a formatter (Black), and understand how to use the debugger. A debugger is not the last resort of someone who is stuck — it is the first tool of someone who understands what their code is doing.

The project

Three projects together — a reproducible data investigation, a Git-first codebase, and a SQL investigation — prove every Foundations competency. A single script does not. Build all three before calling this stage done.

Expected total time: 5–8 weeks alongside the curriculum. Build them roughly in order — each one builds on skills from the previous.

Project 1 — the reproducible data investigation (2–3 weeks)

What it proves: you can find, clean, and analyse real data in Python, with code that actually works from top to bottom.

Choose a messy, interesting public dataset — US government open data portals, the UCI Machine Learning Repository, Kaggle’s free datasets, or any domain you genuinely find interesting. Pick something with at least 5,000 rows and at least 8 columns, including mixed types (numeric, categorical, dates, some missing values). Then execute the full lifecycle:

1. Load and audit. Load the data into a pandas DataFrame. Print shape, dtypes, and value counts for each column. Identify every column with missing values and report what fraction is missing, not just whether any values are missing.
2. Clean — and document every decision.Handle missing values with a justified approach (drop vs. fill vs. leave — state which you chose and why). Detect and investigate outliers using the IQR fence (values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR): report how many there are, look at them, and decide whether they are errors or real extreme values. Fix inconsistent categorical values (e.g. “NYC” vs “New York City” vs “new york”). Document every cleaning step in a comment or markdown cell: what you did and what you would have done differently with more time.
3. Explore. Plot distributions for every numeric variable (histogram + box plot). Plot a correlation heatmap. Identify the three most interesting patterns or surprises in the data. Write one sentence per finding explaining why it might be true.
4. Answer three questions.Frame three specific, answerable questions about the data (“Do cities with higher X have lower Y?” is a question; “explore the data” is not). Use groupby, aggregation, and plots to answer them. Write one paragraph per finding in plain English, as if for a non-technical reader.

The deliverable is a notebook and a README. The notebook must run completely clean from Kernel > Restart & Run All with no errors. The README explains what the dataset is, what question you investigated, and what you found — three paragraphs, no jargon. Publish both to GitHub. The test: hand the link to someone who was not involved and ask them what the project found. If they cannot answer, the README needs work.

Project 2 — the Git-first codebase (1 week)

What it proves: you use version control properly — not as a backup tool but as a collaboration and correctness tool.

Take a small project — a data cleaning script, a simple web scraper, a set of SQL utilities, any code that does something real — and build it entirely through Git-first practices. Requirements:

• At least 20 meaningful commits, each with a clear, conventional commit message (feat:, fix:, refactor:, docs:). A message that says “update” is not a commit message — it is a sign that you are not thinking about the change.
• At least two feature branches merged back to main via a pull request. Even on a solo project, the PR review is the moment you read your own diff critically. Develop the habit now.
• One intentionally simulated merge conflict, resolved cleanly. Understand what a merge conflict is (two branches modified the same lines), how Git marks it, and how to resolve it manually. This is not optional — merge conflicts are inevitable in real work.
• A .gitignore that excludes __pycache__, .env, *.csv data files, and any virtual environment directory. Never commit secrets or large data files.

The deliverable is a public GitHub repository where the commit history tells a coherent story: you can read the log and understand what the project is, how it evolved, and where decisions were made. An interviewer will look at your commit history. Make it say something about how you work.

Project 3 — the SQL investigation (1–2 weeks)

What it proves: you can query a real relational database and communicate findings to a non-technical audience.

Use Google BigQuery’s free public datasets (Chicago taxi trips, Stack Overflow data, US Census — all queryable for free from the console) or download a public SQLite database (Chinook, Northwind, or any multi-table dataset from Kaggle). Answer 10 questions of increasing complexity:

• Basic (3 questions):single-table SELECT with filtering, sorting, and aggregation. “What are the top 10 categories by total revenue?”
• Intermediate (4 questions):JOIN two or more tables, use GROUP BY with HAVING, use at least one subquery. “Which customers have placed more orders than the average customer?”
• Advanced (3 questions):window functions (at least one of: ROW_NUMBER, LAG/LEAD, RANK, running total). “What is the month-over-month growth rate of sales, and which months had growth above 20%?” — requires LAG() and a CTE.

For each question: write the query, explain it in one sentence, and show the result. Then write a one-page plain-English briefing addressed to a non-technical manager: three findings from the data that would actually change a decision. The briefing has no SQL in it — if the finding requires SQL to explain, it is not a finding yet.

Senior extension — automate the reproducibility

Add a Makefile or run.sh to your data investigation project so that anyone can clone the repo, run one command, and reproduce your entire analysis from raw data to final charts. Include dependency pinning (requirements.txt or pyproject.toml). Add a pre-commit hook that runs Black and Flake8 on every commit. This is what “reproducibility” actually means in practice — not “I ran it once and it worked.”

Research-grade extension — understand what you built

For one non-trivial function in your codebase, write a proof sketch: what are the preconditions, what does the function guarantee, and can you construct a case where it fails? This is informal — you are not writing a Coq proof. But the practice of reasoning about correctness rather than just testing with examples is the habit that separates research-grade thinking from production-grade thinking. It starts here.

Resources

Two paths, one goal. CS50x is the long path — rigorous, complete, 10–20 weeks — and worth every hour if you have time. Automate the Boring Stuff is the fast path — useful Python in days. If you already code and just need Python fluency, start with Automate. The Python official tutorial is the reference you consult, not the course you study. Pro Git and the Missing Semester cover the tooling that formal courses always skip. All five are free.

How you know you’re done

Exit criteria — you can answer “yes” to all of these:

• You can write a Python script from scratch — no copy-paste, no Stack Overflow mid-task — that loads a CSV, cleans it, computes summary statistics, and writes the results to a new file.
• You can explain, without looking it up, what a Git branch is, what a commit hash is, and what happens during a merge. You can resolve a merge conflict without panicking.
• Given a SQL prompt and a schema, you can write a query using JOIN, GROUP BY, and at least one aggregate function (SUM, COUNT, AVG) in under five minutes.
• You can navigate the terminal: find files, edit them, run scripts, understand the PATH, and pipe one command’s output into another. You are not afraid of a Permission denied error — you know what it means.
• You understand Big-O at the level of: O(n) is a loop over n items, O(n²) is a loop inside a loop, and you can explain why a dictionary lookup is O(1) while searching a list for a value is O(n).
• An interviewer says “walk me through a coding project you built.” You have a real answer with a real GitHub link, a clear explanation of the problem, and a clean commit history.

Self-test questions

1. Write a Python function that takes a list of integers and returns a new list with all duplicates removed, preserving insertion order. Do not use set() directly. What is the time complexity of your solution?
2. You have a CSV file with 1 million rows. You want to find the top 10 most common values in a specific column. Write the Python code to do this without loading the entire file into memory at once.
3. Without looking it up: what does git rebase main do when run on a feature branch? How does it differ from git merge main?
4. Write a SQL query that finds all customers who placed at least 3 orders in 2024, along with their total order value. What tables and joins would you need?
5. You open a colleague’s Jupyter notebook and run all cells in order. The last cell fails with a NameError: name 'df_clean' is not defined. What likely went wrong, and how do you diagnose it?

Bridge to the next stage

Foundations builds the instrument. Stage 1 puts it to use. The data analyst does not learn new programming concepts — they apply the ones from here at speed, under the pressure of a real question with a real stakeholder waiting for an answer.

What changes at Stage 1 is the mathematics. Statistics — distributions, hypothesis tests, confidence intervals — enters properly, not as a vocabulary word but as a tool for making decisions under uncertainty. The Python you write in Stage 1 is the same Python as here; the thinking behind it is fundamentally different.

What to carry forward:clean reproducible notebooks (non-negotiable from here on), Git as a reflex not a task, SQL fluency, and the discipline of asking “is this code correct” before “does this code run.” These are not revisited. They are assumed.