Week 12: Learning Theory (Baby Steps Towards Symbolic Regression)

DSAN 5300: Statistical Learning
Spring 2026, Georgetown University

Author

Affiliation

Jeff Jacobs

jj1088@georgetown.edu

Open slides in new tab →

Schedule

Today’s Planned Schedule:

	Start	End	Topic
Lecture	6:30pm	7:00pm	Symbolic Regression: What Can a Machine Learn? →
	7:00pm	7:20pm	Comparing Between Groups →
	7:20pm	8:00pm	Regression (Cox Proportional Hazard Model) →
Break!	8:00pm	8:10pm
	8:10pm	9:00pm	Quiz 3 →

source("../dsan-globals/_globals.r")
set.seed(5300)

\[ \DeclareMathOperator*{\argmax}{argmax} \DeclareMathOperator*{\argmin}{argmin} \newcommand{\bigexp}[1]{\exp\mkern-4mu\left[ #1 \right]} \newcommand{\bigexpect}[1]{\mathbb{E}\mkern-4mu \left[ #1 \right]} \newcommand{\definedas}{\overset{\small\text{def}}{=}} \newcommand{\definedalign}{\overset{\phantom{\text{defn}}}{=}} \newcommand{\eqeventual}{\overset{\text{eventually}}{=}} \newcommand{\Err}{\text{Err}} \newcommand{\expect}[1]{\mathbb{E}[#1]} \newcommand{\expectsq}[1]{\mathbb{E}^2[#1]} \newcommand{\fw}[1]{\texttt{#1}} \newcommand{\given}{\mid} \newcommand{\green}[1]{\color{green}{#1}} \newcommand{\heads}{\outcome{heads}} \newcommand{\iid}{\overset{\text{\small{iid}}}{\sim}} \newcommand{\lik}{\mathcal{L}} \newcommand{\loglik}{\ell} \DeclareMathOperator*{\maximize}{maximize} \DeclareMathOperator*{\minimize}{minimize} \newcommand{\mle}{\textsf{ML}} \newcommand{\nimplies}{\;\not\!\!\!\!\implies} \newcommand{\orange}[1]{\color{orange}{#1}} \newcommand{\outcome}[1]{\textsf{#1}} \newcommand{\param}[1]{{\color{purple} #1}} \newcommand{\pgsamplespace}{\{\green{1},\green{2},\green{3},\purp{4},\purp{5},\purp{6}\}} \newcommand{\pedge}[2]{\require{enclose}\enclose{circle}{~{#1}~} \rightarrow \; \enclose{circle}{\kern.01em {#2}~\kern.01em}} \newcommand{\pnode}[1]{\require{enclose}\enclose{circle}{\kern.1em {#1} \kern.1em}} \newcommand{\ponode}[1]{\require{enclose}\enclose{box}[background=lightgray]{{#1}}} \newcommand{\pnodesp}[1]{\require{enclose}\enclose{circle}{~{#1}~}} \newcommand{\purp}[1]{\color{purple}{#1}} \newcommand{\sign}{\text{Sign}} \newcommand{\spacecap}{\; \cap \;} \newcommand{\spacewedge}{\; \wedge \;} \newcommand{\tails}{\outcome{tails}} \newcommand{\Var}[1]{\text{Var}[#1]} \newcommand{\bigVar}[1]{\text{Var}\mkern-4mu \left[ #1 \right]} \]

Symbolic Regression: What Can a Machine Learn?

Thus far, we have either…

• Learned \(X \rightarrow Y\) relationship by setting up a parametric equation like \(Y = \beta_0 + \beta_1X_1 + \cdots + \beta_jX_j\), then figured out how to learn the parameters \(\beta_j\),
• Tried to find a non-parametric way to learn \(X \rightarrow Y\) relationship (\(K\)-Nearest Neighbors! “Memory-based”), or
• Asked the computer to find patterns in \(X\) (no \(Y\)): Unsupervised learning
• There are other (more… “thinking outside the box”) approaches! Let’s look at one (evolutionary computation), then move to thinking more generally about what a machine can learn

Symbolic Regression: “Evolving” Equations

• Think of the relationship \(X \rightarrow Y\) as a mathematical expression with a “DNA” (bitstring) encoding…
• The DNA “nucleotides” (substrings) can code foar:
  • Operations: \(+\), \(-\), \(\times\), \(\div\), \(\sqrt{}\), \(\exp(\cdot)\), \(\sin(\cdot)\), \(\arcsin(\cdot)\), etc.
  • Variables: \(x_1, x_2, \ldots, x_n\)
  • Constants: \(1\), \(−1\), \(0\), \(\pi\), \(e\), \(1.2\)
• Examples: \(y = x_1 + 1\), \(y = x_2\), \(y = x_1 + \sin(x_2)\), \(y = \sin(x_1) + x_2 − 1\), \(y = \sqrt{\sin(x_1) + 1}\)

Slides derived from Andrew Jiang’s very helpful slides!

Evolution of a Computation Graph

From Kronberger et al. (2024)

Ex 1: Learning a Known Equation (Newton)

Figure 1 from de Silva et al. (2020)

Figure 4 from de Silva et al. (2020)

The Learned Equation

SR Result:

\[ \begin{aligned} \text{distance} = &4.2026 t^2 + 0.0177 m^2 + 0.5456 tm \\ &− 0.006564 ct - 4.2614 \times 10^{-4} cm \\ &+ 2.5633 \times 10^{-6} c^2 \end{aligned} \]

From Newton’s Laws:

\[ \text{distance} = \frac{g}{2}t^2 \]

SR-learned equation agreement with (a) Data and (b) Newton’s Laws, from Kronberger et al. (2024)

Example 2: Finding an “Approximately Good” Equation

Table 1: An evolved relationship, from Schmidt and Lipson (2009)

Accuracy	Equations in Sequence	Event
-1.4197	\(x + x - c_3 - y\)	random
-1.41347	\(x + x + x - c_4 - y\)	mutation
-1.41339	\(x + x + x - \sin(c_3) - y\)	mutation
-1.13805	\(x + x + x - \sin(y) - (x - x)\)	crossover
-1.08904	\((x + x)\cdot x - \sin(y) - (x - x)\)	mutation
-1.08574	\((x + x)\cdot x - \sin(y) - c_1\)	mutation
-1.01841	\((x + x)\cdot x - y - c_1\)	mutation
-0.978484	\((x + x + x)\cdot x - y - c_{13}\)	mutation
-0.914336	\((x + y - c_3)\cdot y + x\cdot x \cdot c_{15}\)	mutation
-0.303559	\((x + y - c_4)\cdot y + x \cdot x \cdot c_{15}\)	mutation
-0.0692607	\((x + y - \sin(x))·y + x \cdot x \cdot c_{15}\)	crossover
-0.0140815	\((x + y - x)·y + x·x·c_{15}\)	mutation
-0.0050732	\((x + y - x)·y + x·x·c_{16}\)	mutation
-0.0050732	\(y \cdot y + c_3 \cdot x \cdot x\)	mutation

From Wolfram Blog

The Takeaway

• If you have a sense for the “pieces” of the solution, but not how to put them together… a genetic algorithm might be for you!
• Evolving silly cars
• Evolving music 😉

Statistical Learning Theory

PAC Learning

Quiz Time!

Quiz 12

References

Kronberger, Gabriel, Bogdan Burlacu, Michael Kommenda, Stephan M. Winkler, and Michael Affenzeller. 2024. Symbolic Regression. 1st ed. Chapman and Hall/CRC. https://doi.org/10.1201/9781315166407.

Schmidt, Michael, and Hod Lipson. 2009. “Distilling Free-Form Natural Laws from Experimental Data.” Science 324 (5923): 81–85. https://doi.org/10.1126/science.1165893.

Silva, Brian M. de, David M. Higdon, Steven L. Brunton, and J. Nathan Kutz. 2020. “Discovery of Physics From Data: Universal Laws and Discrepancies.” Frontiers in Artificial Intelligence 3 (April). https://doi.org/10.3389/frai.2020.00025.