Numbers, Letters, Sometimes Both

Calibration of Mind Changing for Bayesian AB Tests

Bayesian decision making with loss functions was a very popular with stakeholders and data scientists when I was working in AB testing. In short, eschew p values instead ask…
Sep 1, 2024
Demetri Pananos

What Is Confounding?

Thinking back to my Introduction to Epidemiology class, confounding was one of the topics of central interest. However, it took the remainder of my PhD (and then some) to…
Jun 26, 2024
Demetri Pananos

Interesting Interview Questions

Probability
I’m about to do some interviews this week which got me reflecting on some of my favorite questions I’ve been asked. Usually, these are little toy problems you’d find in an…
May 21, 2024
Demetri Pananos

Peeking Sets You Up For Dissapointment

Peeking (looking for significance in an AB test before the experiment has enough samples to reach desired power) is a “no no”. Rationales for not peeking typically mention…
Feb 8, 2024
Demetri Pananos

You Just Said Something Wrong About Logistic Regression

Congratulations, you just said something wrong about logistic regression. That’s OK, logistic regression is hard and we all have to learn/re-learn some things from time to…
Feb 6, 2024
Demetri Pananos

The Winner’s Curse Is Easy To Understand From This Picture

Take a look at the photo below, and it i should be easy to understand why The Winner’s Curse (the general tendency for detected effects to be an over estimate of the truth)…
Jan 20, 2024
Demetri Pananos

Survival Analysis With Logistic Regression

This question came up on cross validated. In the course of responding to my answer, OP asks “why can’t I adjust for exposure length in a logistic regression” which kind of…
Jan 20, 2024
Demetri Pananos

The Generalized Gamma Distribution For Parametric Survival

A couple weeks back, I posted a little something something on the Shifted Beta Geometric distribution. That distribution is used in estimating churn in contractual settings…
Dec 2, 2023
Demetri Pananos

Brain Teaser About Frequentist Statistics

Ron Kohavi is somewhat of a prominent figure in the A/B testing community. Armed with having coauthored “Trustworthy Online Controlled Experiments”, experience at places…
Dec 1, 2023
Demetri Pananos

Estimating the Shifted Beta Geometric Model in Stan

The shifted beta geometric model (sBG) is a model that is used to forecast retention/survival of users in contractual settings (think netflix, disney plus, tinder gold…
Nov 20, 2023
Demetri Pananos

Regularization of Noisy Multinomial Counts

I’m reading Statistical Learning with Sparsity. Its like Elements of Statistical Learning, but just for the LASSO. In chapter 3, there is this interesting example of using…
Sep 24, 2023
Demetri Pananos

 

Right In Ways You Don’t Care About

I’ve typically been long on the value of data scientists but I think I’m starting to lose my optimism.
Sep 11, 2023
Demetri Pananos

Difference in Difference Estimates Can Be Inaccurate When Adjusting For Detected Pre-Treatment Differences in Randomized Experiments

AB Testing
Statistics
Causal Inference
Edits:
Sep 3, 2023
Demetri Pananos

 

Did You Do Your Homework?

This tweet made its way to my corner of the Bird Site. I’m a glutton for math homework punishment so I attempted it (and failed) but do not confuse my interest as…
Jun 26, 2023
Demetri Pananos

 

A Practical A/B Testing Brain Dump

AB Tests
Statistics
Over the past year or so, I’ve helped move Zapier towards doing better experiments and more of them. I wanted to document some of this for posterity; what I did, what…
May 14, 2023
Demetri Pananos

Choosing the Optimal MDE for Experimentation

AB Tests
Statistics
Planning experiments means deciding how long to run the experiment (by doing a sample size calculation). A big factor in this decision is agreeing on a minimal detectable…
Mar 31, 2023
Demetri Pananos

Causal Risk Ratios in AB Tests with One Sided Non-Compliance

AB Tests
Statistics
Causal Inference
Sometimes we run AB tests where engaging with the treatment (and hence being treated) is optional. Since assignment to treatment is random, we can use the randomization as…
Feb 11, 2023
Demetri Pananos

 

Journeyman Statistics

A number of people have asked me how to learn statistics. I don’t have a good answer for them. I find all books are deficient in some way: some too theoretical, others not…
Dec 31, 2022
Demetri Pananos

 

Way Too Many Taylor Series

Statistics
AB Testing
Let \(Y(a)\) be a potential outcome for a continuous measure under treatment status \(A=a\), which for the purpose of this blog post can be considered a binary treatment.…
Nov 25, 2022
Demetri Pananos

Bootstrapping in SQL

Statistics
Remember the “Double Down” from KFC? It was bacon and cheese sandwiched between two deep fried pieces of chicken. I’m willing to bet we all conceived of it independently (as…
Nov 16, 2022
Demetri Pananos

PCA on The Tags for Cross Validated

Machine Learning
Julia Silge gave a really good talk in 2018 about PCA ond tags on stack overflow. She was able to interpret some of the components to infer some subgroups of users of stack…
Aug 16, 2022
Demetri Pananos

Interim Analysis & Group Sequential Designs Pt 2

Statistics
AB Testing
This is part 2 of an ongoing series on group sequential designs for AB testing. Previous parts are shown below
Jul 30, 2022
Demetri Pananos

Forecasting Experimental Lift Using Hierarchical Bayesian Modelling

Statistics
AB Testing
Bayes
You’re part of a team at a company who is tasked with improving conversion on some web page. You’ve run a few experiments already with mixed results and now it is time to…
Jul 20, 2022
Demetri Pananos

Interim Analysis & Group Sequential Designs Pt 1

Statistics
AB Testing
Special thanks to Jacob Fiksel for writing a great blog post which inspired me to write my own.
Jul 6, 2022
Demetri Pananos

This Is A Quarto Blog

News
This is a quarto blog.
Jun 22, 2022
Demetri Pananos

Flippin’ Fun!

Bayes
Stan
Python
I wrote an answer to a question about sequences of coin flips a couple days back that I was quite chuffed with. In short, the question asked for statistical ways to…
May 7, 2022
Demetri Pananos

 

Hacking Sklearn To Do The Optimism Corrected Bootstrap

Statistics
Machine Learning
Python
Scikit-Learn
Its late, I can’t sleep, so I’m writing a blog post about the optimism corrected bootstrap.
Nov 23, 2021
Demetri Pananos

 

On Interpretations of Confidence Intervals

Statistics
The 95% in 95% confidence interval refers not to the probability that any one interval contains the estimand, but rather to the long term relative frequency of the estimator…
Apr 3, 2021
Demetri Pananos

 

Log Link vs. Log(y)

R
Statistics
You wanna see a little gotcha in statistics? Take the following data
Oct 6, 2020
Demetri Pananos

Gradient Descent with ODEs

Python
Machine Learning
Statistics
Gradient descent usually isn’t used to fit Ordinary Differential Equations (ODEs) to data (at least, that isn’t how the Applied Mathematics departments to which I have been…
May 21, 2019
Demetri Pananos

 

Neat Litle Combinatorics Problem

Python
Probability
I’ll cut right to it. Consider the set \(S = (49, 8, 48, 15, 47, 4, 16, 23, 43, 44, 42, 45, 46 )\). What is the expected value for the minimum of 6 samples from this set?
Aug 31, 2018
Demetri Pananos

Coins and Factors

Python
Riddler
I love Fivethirtyeight’s Riddler column. Usually, I can solve the problem with computation, but on some rare occasions I can do some interesting math to get the solution…
Dec 29, 2017
Demetri Pananos
No matching items