Published in Towards Data Science·Feb 26Use simulations to optimize customer wait time, systems load, and costExperiments are too expensive? Use simulations instead. — Let’s say your infrastructure is a fleet of M identical machines, each processing a single task at a time. Each new customer is placed in a queue waiting for a machine to become idle. You know how long each processing task takes to complete, but it’s not a fixed number…Data Science13 min read
Published in Towards Data Science·Aug 29, 2021Data has many periodic components you need to visualize? Treat it like audio!Visualize periodic components using the Fourier transform. — You’re working with timeseries data, and you’re at the initial discovery stage. You’re trying to figure out all the different ways that the data varies in time. There may be constant trends, whereby the data is constantly increasing or decreasing in time. There may be completely random variations. And there…Fourier Transform7 min read
Published in Towards Data Science·Jul 15, 2021Backpropagation from Scratch: How Neural Networks Really WorkHow do neural networks really work? A complete example, written from scratch in Python, with all the math, explained in plain language. — How do neural networks really work? I will show you a complete example, written from scratch in Python, with all the math you need to completely understand the process. I will explain everything in plain English as well. You could just follow along, read just the text and still get…Neural Networks16 min read
Published in Towards Data Science·Jan 14, 2021Bootstrap statistics — how to work around limitations of simple statistical testsYou have a data sample. From it, you want to calculate a confidence interval for the population mean value. What’s the first thing you… — You have a data sample. From it, you want to calculate a confidence interval for the population mean value. What’s the first thing you think about? It’s usually a t-test. But the t-test has several requirements, one of which is that the sampling distribution of the mean is nearly normal…Data Science7 min read
Published in Towards Data Science·Dec 28, 2020Pfizer and Moderna — vaccine efficacy calculated from dataCOVID-19 vaccine efficiency calculated from data with statistics — But how was that number calculated? Turns out, the basic value is pretty easy. I’ll show you how to do that, and then I will make an estimate for how confident we are that the value is right. Randomized trials To test a vaccine, you need to do a randomized blind trial…Statistics6 min read
Oct 31, 2020Test results, part 2: how about negative results?Bayes’s Theorem from statistics applied to pandemics — This is part 2 of this article: Test results are positive. What are the odds you have the virus? Surprise, intuition is wrong! Bayes’s Theorem from statistics applied to pandemicsflorin-andrei.medium.com To recap: there’s a pandemic going on, 1% of people have the virus. There’s a test that can detect the virus, and the test is 99% reliable (for both positive and negative results).Statistics2 min read
Oct 30, 2020Test results are positive. What are the odds you have the virus? Surprise, intuition is wrong!Bayes’s Theorem from statistics applied to pandemics — Let’s say there’s a virus pandemic sweeping through the population, and 1% of people have the virus. Let’s say there’s a test for this condition, and the test is 99% reliable, meaning — out of 100 tested cases, the test will be correct in 99 cases, and will be wrong…Statistics5 min read
Published in Towards Data Science·Jun 21, 2020Removing non-linear trends from timeseries dataDetrending, polynomial regression — a step up from linear regression. — Sometimes trends need to be removed from timeseries data, in preparation for the next steps, or part of the data cleaning process. If you can identify a trend, then simply subtract it from the data, and the result is detrended data. If the trend is linear, you can find it…Regression5 min read
Published in Towards Data Science·Jun 7, 2020The potato train — using Python with extremely large numbers and arbitrary precision for binomial probabilityHow to deal with very large numbers in Python, using the binomial distribution from statistics as an example — Math is hard, let’s go shopping — for tutorials, that is. I definitely wish I had read this tutorial before trying some things in Python that involve extremely large numbers (binomial probability for large values of n) and my code started to crash. But wait, I hear you saying, Python…Python9 min read
