Not All Predictive Analytics Are Created Equal: Watch Out for These 4 Potential Pitfalls

Posted by Steve SmithO

August 1, 2019

Predictive analytics are transforming education for the better, but many districts encounter difficulties when they attempt to scale their analytics solutions. Here are four common pitfalls for which to look out.

Predictive analytics are designed to help organizations predict potential outcomes, enabling key decision-makers to address issues sooner than previously possible. Although this technology sounds too good to be true, it is in fact very real, and many organizations across various industries are already reaping its rewards.

In the education space, district leaders can use predictive analytics to improve student outcomes in a variety of ways. From identifying which students are at risk of not passing their exams to improving early intervention practices to gaining insight into which students are unlikely to graduate several years down the line, predictive analytics are revolutionizing the way educators help students succeed.

Unfortunately, scaling predictive analytics solutions can be difficult, and many districts struggle to realize their solutions’ full potential. As school districts across the country set out to implement predictive solutions of their own, it will be important for them to keep these common pitfalls of predictive analytics in mind.

1. Low-Quality Data Inputs

Predictive analytics leverage longitudinal data to predict future events. Thus, if you use poor data, it stands to reason that your predictions will be poor as well. To make accurate predictions, algorithms must be fed clean, validated, error-free data. Additionally, the data will have to be “preprocessed.” This includes instances where the data have missing values or values that must be transformed (e.g., from a 0-100 scale to a four-point GPA, from assessments to Z scores, etc.). For many machine learning platforms, missing values must either be dropped or imputed.

Most of the generic predictive modeling tools on the market generate predictions with user defined data inputs. If the user providing data to the software does not clean and preprocess the data correctly or consistently, results can be difficult to interpret, misleading, or even inaccurate.

To avoid these concerns, a school district must perform tedious data cleansing and transformation processes and perhaps even data imputations. Unfortunately, few predictive analytics providers will partner with schools to help them through the data cleansing process. Consequently, district leaders are forced to hire third parties to help them examine their data’s health and ensure their data inputs are consistent with one another. While such relationships can function as an adequate stopgap measure, adding additional vendor relationships is less than ideal since they can be time-consuming and expensive.

To set our partners up for success, Hoonuit provides end-to-end data management infrastructure that facilitates data transformation and cleansing. Our software engineers have built a matching process that ensures records from disparate data sources are connected regardless of their field setup. We also run cross-domain analyses and blend multiple-source data to help educators maximize the value of their analytics solutions. Moreover, our data scientists ensure appropriate methods are used for scaling/rescaling assessments, multivariate imputation for missing data, modeling data relationships and specific use of ML algorithms, and managing the data quality, accuracy, and usability of machine learning outputs for dashboard displays and general data consumption.

2. A Lack of Longitudinal Insight

Access to rich longitudinal data is an integral piece of the predictive analytics puzzle. If you want to make accurate predictions about the future, you must model the relationships between inputs and observed outcomes. If, for instance, you want to predict 6th grade test scores based on 5th grade test scores, you would need to build a cohort of students that took both the 5th and 6th grade tests. Modeling the relationship between these data is referred to as “training.”

Most generic predictive tools ingest the data housed within districts’ data warehouses without first vetting it for missing information or evaluating how far back the data go. If districts are not alerted to gaps in their data, they will continue to feed incomplete datasets to their tools, and, more often than not, will be unsatisfied with the quality (or lack) of the resulting predictions.

To improve the accuracy of their tools’ predictions, districts must invest in solutions that pair sophisticated analytics with robust data warehousing capabilities. This combination helps districts streamline the prediction-making process from the initial data collection stage all the way through to the analysis stage. It also allows districts to store and organize many years’ worth of data — some districts using Hoonuit have nearly 20 years’ worth of student data stored in our warehouses!

By acting as both a data warehouse and a predictive analytics provider, we can say with confidence that our predictive solution will help your district get the best results out of your student data.

3. Insufficient Domain Expertise

Machine learning algorithms perform best when they are crafted according to an underlying theory or are supported by previous research. Predictive models built solely on accuracy can often seem like a black box, and are less likely to be institutionalized or gain widespread adoption.

Unfortunately — but perhaps not unsurprisingly — generic predictive analytics tools are not built specifically for educators. As a result, it falls to district data leaders to adapt their solutions to the education context by choosing the correct inputs, implementing data preprocessing, and interpreting algorithmic outputs. This can become very complicated, very quickly.

Partnering with Hoonuit lifts this burden from district stakeholders’ shoulders. Our data scientists have extensive experience in the education space, so we have first-hand knowledge of the best practices for predicting educational outcomes. Further, we have an ongoing development cycle through which we are continuously testing and iterating to ensure districts are getting the most out of our solution’s predictive capabilities.

4. Training Multiple Algorithms

Many generic predictive analytics solutions only allow you to use one machine learning algorithm to generate predictions. While this single-track approach simplifies the process for users, it can also lead to less accurate predictions. Additionally, some algorithms have a tendency to perform better when used with certain types of data inputs. If you are limited to a generic pre-packaged algorithm, you could potentially be generating less accurate predictions without knowing it.

At Hoonuit, we generate predictions using a minimum of five algorithms. After evaluating the contextual accuracy of each, we are able to automatically select the most accurate algorithm for each or our clients. What’s more, since each algorithm learns independently of the others, the accuracy of our predictions becomes better with time (or, more precisely, with exposure to more longitudinal data).

While there are other solutions that use multiple algorithms to generate predictions, they place the onus on educators to decide which algorithm to use. This poses an immense challenge for tech-unsavvy users who might not understand how to pick — let alone optimize — the right algorithm.

Hoonuit works closely with educators to help them decide which algorithm is best for any given set of circumstances. We collaborate with you to fine-tune your predictive algorithms for specific datasets, so you never have to worry about wading into the complex mathematics that are always running “behind the scenes.”

The Hoonuit Solution

At Hoonuit, we understand that every district has its own challenges and goals, and we believe that you shouldn’t have to figure out the way forward all on your own. As your partner, we work with you on initiatives ranging from earlier at-risk identification to intervention improvement to increased two- and four-year postsecondary placement. With a partner like Hoonuit on your side, cutting-edge predictive analytics can help you improve and evolve on every front.

Being aware of the pitfalls outlined above is step one toward realizing this potential for improvement and evolution. The next step is ensuring your analytics tools are easily accessible and ready-to-use, which is why Hoonuit provides tools that are as user-friendly and intuitive as they are effective.

Learn more about our solution today!

Get in touch

Get in touch