Reading programming error messages is the first step to understanding what it is trying to tell the programmer about how to fix errors in their code. However, these are often difficult to read, especially for novices which is not surprising given that error messages in many of the most popular languages in which novices learn to code were not written with readability in mind. As a result, novices frequently struggle to understand them. This is a long-standing problem, with researchers highlighting concerns around Programming error message readability over the last six decades. Very recent work has finally put forward evidence of the need for measuring readability in error messages and a framework for doing so. This framework consists of four important pillars of readability for Programming error messages: message length, vocabulary, jargon, and sentence construction. We use this framework to propose an approach that could be used to automatically assess the readability of Programming error messages in any language. Using the established readability factors as predictors in a machine learning model, we train several models using a dataset of messages from C and Java. We examine the performance of these models, and apply the best performing model to a previously published set of messages evaluated for readability by experts, non-experts and students. Our results validate the previously proposed four factors, and our model classifies messages similarly to human raters. Finally, we discuss ideas for future work that are needed to improve the accuracy of the model.