Data science today is a lot like the Wild West: there’s endless opportunity and excitement, but also a lot of chaos and confusion. If you’re new to data science and applied machine learning, evaluating a machine-learning model can seem pretty overwhelming. Now you have help. With this O’Reilly report, machine-learning expert Alice Zheng takes you through the model evaluation basics.
In this overview, Zheng first introduces the machine-learning workflow, and then dives into evaluation metrics and model selection. The latter half of the report focuses on hyperparameter tuning and A/B testing, which may benefit more seasoned machine-learning practitioners.
With this report, you will:
- Learn the stages involved when developing a machine-learning model for use in a software application
- Understand the metrics used for supervised learning models, including classification, regression, and ranking
- Walk through evaluation mechanisms, such as hold?out validation, cross-validation, and bootstrapping
- Explore hyperparameter tuning in detail, and discover why it’s so difficult
- Learn the pitfalls of A/B testing, and examine a promising alternative: multi-armed bandits
- Get suggestions for further reading, as well as useful software packages
O'Reilly Media, Inc.
O'Reilly Media spreads the knowledge of innovators through its books, online services, magazines, research, and conferences. Since 1978, O'Reilly has been a chronicler and catalyst of leading-edge development, homing in on the technology trends that really matter and galvanizing their adoption by amplifying "faint signals" from the alpha geeks who are creating the future. An active participant in the technology community, the company has a long history of advocacy, meme-making, and evangelism.