Numbers, Letters, Sometimes Both – Demetri Pananos Ph.D

Adaptive Shrinkage and the Student’s t Prior

AB Testing

Bayes

Statistics

R

Before we get started, I want to say that here may be good reasons to use a normal prior over a Student’s t prior. Statistics may be the science of defaults, but I am not…

Demetri Pananos

Evolving Thoughts on AI and Academic Writing

Misc

Career

The “Text” section of my “How I Use AI” reads

Demetri Pananos

Better Priors for Bayesian Analysis of A/B Testing

AB Testing

Bayes

Statistics

When estimating a treatment effect (typically the lift), \(\delta_i\), using the Bayesian methodology, Datadog uses a default prior of

Demetri Pananos

Adventures in Covariance

Statistics

Machine Learning

Let \(\mathbf{x}_1, \cdots, \mathbf{x}_n\) be a sample of \(n\) dimensional random variables with mean \(\mu\). Their covariance is defined as

Demetri Pananos

What Do You Do When There Are No More Backyards To Play In?

Misc

Career

I work 10 hours per week at my University’s medical school consulting on statistics for residents and physicians. I’ve noticed far fewer requests for work, and I think AI is…

Demetri Pananos

Musings About The Normal Normal Model

Statistics

Bayes

Stan

R

In accordance with my AI policy, I have written this blog post myself. However, all code is written Claude and a lot of the mathematics is done in conjunction with Gemini.

Demetri Pananos

Optimal Experiment Sample Sizes for Bayesian A/B Tests

AB Testing

Bayes

Statistics

This post is a spiritual successor to an early post on this blog and a fast follow here from this post about A/B test run times for Bayesians. From that post, we are able to…

Demetri Pananos

How Long To Run an A/B Test as a Bayesian

AB Testing

Bayes

Statistics

This blog post is inspired, almost entirely, by Tyler Buffington’s excellent statistics reading group presentation today at Datadog on the value of information in A/B…

Demetri Pananos

Things I Believe According to Claude

Misc

Career

I asked claude code to read through all the posts I’ve made in this blog in order to pull out some of my principles. Here is what claude thinks I believe

Demetri Pananos

Cleaning Data Patterns

R

Misc

I still clean data from time to time, and I find myself writing patterns that looks like

Demetri Pananos

Kuethe’s Criterion and A/B Tests

AB Testing

Statistics

A/B tests often target the “lift” in the metric as an estimand. As a reminder, the lift is

Demetri Pananos

How to Fit a Generalized Linear Model with Fixed Effects (Pt 1)

Statistics

Regression

R

To the Economists, I am sorry.

Demetri Pananos

On Climbing Hills

Misc

Career

I’ve been climbing this hill for what feels like a long time. I can look back and see how high I’ve climbed, the rocks I’ve had to overcome, and the gullies I nearly was…

Demetri Pananos

Pre Randomized Experiment Differences Don’t Mean DiD is Needed

AB Testing

Statistics

Causal Inference

Take a look at this interaction I had with user jginestet on Cross Validated. In short, jginestet suggests that a randomized experiment assessing percent changes in tumor…

Demetri Pananos

Let’s Take About 15% Of The Top There

Statistics

Bayes

R

I keep coming back to A New Look at P Values for Randomized Clinical Trials – an instant classic paper on RCTs, which could be extended to online experimentation as well.…

Demetri Pananos

More on Bayesian Statistics

Statistics

Bayes

I recently wrote a blog post for my employer in which I made some claims regarding Bayesian statistics.

Demetri Pananos

A Brief Tour of CUPED and Related Methods (Pt. 1)

AB Testing

Statistics

Regression

Randomized experiments have a rich history, almost none of which is the concern of this post. Rather, I want to focus on a tiny slice of randomized experiment literature…

Demetri Pananos

Calibration of Mind Changing for Bayesian AB Tests

AB Testing

Statistics

Bayes

Bayesian decision making with loss functions was a very popular with stakeholders and data scientists when I was working in AB testing. In short, eschew p values instead ask…

Demetri Pananos

What Is Confounding?

Statistics

Causal Inference

R

Thinking back to my Introduction to Epidemiology class, confounding was one of the topics of central interest. However, it took the remainder of my PhD (and then some) to…

Demetri Pananos

Interesting Interview Questions

Probability

Misc

I’m about to do some interviews this week which got me reflecting on some of my favorite questions I’ve been asked. Usually, these are little toy problems you’d find in an…

Demetri Pananos

Peeking Sets You Up For Dissapointment

AB Testing

Statistics

Peeking (looking for significance in an AB test before the experiment has enough samples to reach desired power) is a “no no”. Rationales for not peeking typically mention…

Demetri Pananos

You Just Said Something Wrong About Logistic Regression

Statistics

Regression

Congratulations, you just said something wrong about logistic regression. That’s OK, logistic regression is hard and we all have to learn/re-learn some things from time to…

Demetri Pananos

The Winner’s Curse Is Easy To Understand From This Picture

AB Testing

Statistics

R

Take a look at the photo below, and it i should be easy to understand why The Winner’s Curse (the general tendency for detected effects to be an over estimate of the truth)…

Demetri Pananos

Survival Analysis With Logistic Regression

Statistics

Survival Analysis

Regression

R

This question came up on cross validated. In the course of responding to my answer, OP asks “why can’t I adjust for exposure length in a logistic regression” which kind of…

Demetri Pananos

The Generalized Gamma Distribution For Parametric Survival

Statistics

Survival Analysis

Bayes

Stan

R

A couple weeks back, I posted a little something something on the Shifted Beta Geometric distribution. That distribution is used in estimating churn in contractual settings…

Demetri Pananos

Brain Teaser About Frequentist Statistics

AB Testing

Statistics

R

Ron Kohavi is somewhat of a prominent figure in the A/B testing community. Armed with having coauthored “Trustworthy Online Controlled Experiments”, experience at places…

Demetri Pananos

Estimating the Shifted Beta Geometric Model in Stan

Statistics

Bayes

Stan

Survival Analysis

R

The shifted beta geometric model (sBG) is a model that is used to forecast retention/survival of users in contractual settings (think netflix, disney plus, tinder gold…

Demetri Pananos

Regularization of Noisy Multinomial Counts

Statistics

Machine Learning

R

I’m reading Statistical Learning with Sparsity. Its like Elements of Statistical Learning, but just for the LASSO. In chapter 3, there is this interesting example of using…

Demetri Pananos

Right In Ways You Don’t Care About

Misc

Career

I’ve typically been long on the value of data scientists but I think I’m starting to lose my optimism.

Demetri Pananos

Difference in Difference Estimates Can Be Inaccurate When Adjusting For Detected Pre-Treatment Differences in Randomized Experiments

AB Testing

Statistics

Causal Inference

Demetri Pananos

Did You Do Your Homework?

Misc

Career

This tweet made its way to my corner of the Bird Site. I’m a glutton for math homework punishment so I attempted it (and failed) but do not confuse my interest as…

Demetri Pananos

A Practical A/B Testing Brain Dump

AB Testing

Statistics

Over the past year or so, I’ve helped move Zapier towards doing better experiments and more of them. I wanted to document some of this for posterity; what I did, what…

Demetri Pananos

Choosing the Optimal MDE for Experimentation

AB Testing

Statistics

Planning experiments means deciding how long to run the experiment (by doing a sample size calculation). A big factor in this decision is agreeing on a minimal detectable…

Demetri Pananos

Causal Risk Ratios in AB Tests with One Sided Non-Compliance

AB Testing

Statistics

Causal Inference

Sometimes we run AB tests where engaging with the treatment (and hence being treated) is optional. Since assignment to treatment is random, we can use the randomization as…

Demetri Pananos

Journeyman Statistics

Statistics

Misc

A number of people have asked me how to learn statistics. I don’t have a good answer for them. I find all books are deficient in some way: some too theoretical, others not…

Demetri Pananos

Way Too Many Taylor Series

Statistics

AB Testing

Let \(Y(a)\) be a potential outcome for a continuous measure under treatment status \(A=a\), which for the purpose of this blog post can be considered a binary treatment.…

Demetri Pananos

Bootstrapping in SQL

Statistics

SQL

Remember the “Double Down” from KFC? It was bacon and cheese sandwiched between two deep fried pieces of chicken. I’m willing to bet we all conceived of it independently (as…

Demetri Pananos

PCA on The Tags for Cross Validated

Machine Learning

R

Julia Silge gave a really good talk in 2018 about PCA ond tags on stack overflow. She was able to interpret some of the components to infer some subgroups of users of stack…

Demetri Pananos

Interim Analysis & Group Sequential Designs Pt 2

Statistics

AB Testing

This is part 2 of an ongoing series on group sequential designs for AB testing. Previous parts are shown below

Demetri Pananos

Forecasting Experimental Lift Using Hierarchical Bayesian Modelling

Statistics

AB Testing

Bayes

You’re part of a team at a company who is tasked with improving conversion on some web page. You’ve run a few experiments already with mixed results and now it is time to…

Demetri Pananos

Interim Analysis & Group Sequential Designs Pt 1

Statistics

AB Testing

Special thanks to Jacob Fiksel for writing a great blog post which inspired me to write my own.

Demetri Pananos

This Is A Quarto Blog

News

This is a quarto blog.

Demetri Pananos

Flippin’ Fun!

Statistics

Bayes

Stan

Python

I wrote an answer to a question about sequences of coin flips a couple days back that I was quite chuffed with. In short, the question asked for statistical ways to…

Demetri Pananos

Hacking Sklearn To Do The Optimism Corrected Bootstrap

Statistics

Machine Learning

Python

Scikit-Learn

Its late, I can’t sleep, so I’m writing a blog post about the optimism corrected bootstrap.

Demetri Pananos

On Interpretations of Confidence Intervals

Statistics

The 95% in 95% confidence interval refers not to the probability that any one interval contains the estimand, but rather to the long term relative frequency of the estimator…

Demetri Pananos

Log Link vs. Log(y)

R

Statistics

Regression

You wanna see a little gotcha in statistics? Take the following data

Demetri Pananos

Gradient Descent with ODEs

Python

Machine Learning

Statistics

Gradient descent usually isn’t used to fit Ordinary Differential Equations (ODEs) to data (at least, that isn’t how the Applied Mathematics departments to which I have been…

Demetri Pananos

Neat Litle Combinatorics Problem

Python

Probability

I’ll cut right to it. Consider the set \(S = (49, 8, 48, 15, 47, 4, 16, 23, 43, 44, 42, 45, 46 )\). What is the expected value for the minimum of 6 samples from this set?

Demetri Pananos

Coins and Factors

Python

Probability

Riddler

I love Fivethirtyeight’s Riddler column. Usually, I can solve the problem with computation, but on some rare occasions I can do some interesting math to get the solution…

Demetri Pananos