Awesome article, is so good to hear from people who really know so well what they are doing! Thanks for sharing this knowledge with us. I only have one question about when to decide to keep or not an experiment. If after a test A/B the treatment group doesn't show any meaningful evidence why did you guys decided to keep the new experience? Like was said, sometimes new features have bugs, and I think maybe is harder to keep and maintain new code. If we start thinking about possible bugs, regressive testing changes that should be made, documentation, scalability, and so on. Wouldn't it be better to keep the original experiment if the test doesn't materialize on any evidence?

Caíque Coelho

A passionate developer, working as QA Engineer at QuintoAndar, also an App Developer on weekends and a Data Scientist on free time. Founder App Teste Eneagrama.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store