Clever Engineering Blog — Always a Student

Engineering at Clever

By Rafael Garcia on

At Clever we’re building a way for students and teachers to start using learning applications with a click of a button. This is incredibly difficult to do in a school environment, because existing infrastructure is typically incompatible with a world where students use software on the internet.

The backbone of the infrastructure at most schools is the student information system (SIS), which stores basic information about students, teachers, and classes.
The SIS serves many purposes, but integrating with other software is not one of them. For example, if Oakland Unified School District wants to give each of its 32,000 students an email address, it either needs to manage this by hand or hire expensive technical consultants to write a custom integration.

Each additional piece of software only magnifies this burden. The end result is that the ease and convenience people expect when adopting new technology in their personal lives–whether downloading an app onto their phone or logging into something on the internet–do not exist within education.

So what does it take to build a platform for students and teachers to easily access an increasing number of great learning applications? The rest of this post goes into some of the technical challenges we face building such a platform at Clever.

One API for thousands of legacy systems

The Clever API is essentially a drop-in infrastructure upgrade for schools that enables their systems to start communicating with the applications they want to use.
For application developers, we take care of the integration with legacy SISes so they can focus on building a great learning experience. These two challenges–keeping the school setup process simple and providing a common developer interface to the fragmented world of school data systems–prove difficult at Clever’s scale.

A unique “big data” problem

Every hour Clever syncs data from 20,000 schools with the various applications they use. Though the data synced is fairly basic, this translates into over a gigabyte of data synced per minute–well over a terabyte per day. Additionally, to make data sync easier for developers we provide an event stream of diffs in the API. Creating a data pipeline that efficiently syncs and diffs a large (and growing) amount of data around the clock is a unique problem to which we’ve had to find creative solutions.

Availability

In less than two years, Clever has come to be used in over 1 in 6 schools, making it one of the fastest-growing companies in education. Our current scale and continued growth present a constant stream of challenges regarding scalability and availability. Additionally, as more students and teachers use Clever to log in to applications, Clever downtime translates into costly classroom downtime.

Security and privacy

Before Clever, schools typically sent data to applications via very insecure methods, e.g. emailing CSVs back and forth. Clever not only modernizes this process, but also provides schools a large amount of flexibility regarding the kind of authorizations they grant to applications. Giving schools fine-grained tools to control data permissions presents additional challenges in providing a high-performance API that correctly scopes data it returns.

This is just a sample of the technical challenges we face at Clever. Stay tuned to this blog as we dive deeper into some of the ways we go about solving these problems, and if you’re an engineer passionate about technology and education, join us!

Rafael Garcia