This repository contains the code to reproduce the experiments reported in (Metelli et al., A tale of sampling and estimation in discounted reinforcement learning, 2022) published at AISTATS 2023. The paper can be found at https://proceedings.mlr.press/v206/metelli23a.html.