Reducing your labeled data requirements (2–5x) for Deep Learning: Deep Mind’s new “Contrastive Predictive Coding 2.0”

CPC 2.0 in action — with only 1% of labeled data, achieves 74% accuracy (from the paper)

Current Deep Learning for vision, audio, etc. requires vast amounts of human labeled data, with many examples of each category, to properly train a classifier to acceptable accuracy.

By contrast, humans only need to see a few examples of a class to begin properly and accurately recognizing and classifying future examples of that class.

The difference is that humans are able to quickly generate accurate mental ‘representations’ of things, and then use those representations to flexibly account for future variations. After seeing a few images of bluejays for example, we can make a mental model or representation of a bluejay, and then spot and accurately id bluejays in new images even when the birds are facing different angles, different perspectives, etc. Deep learning struggles to build representations in the same manner, and thus needs to train on lots and lots and lots of labeled instances (with images ‘augmented’ to show different angles, perspectives) to reliably handle future data and successfully generalize.

That gap in representation ability, which drives the requirement for large amounts of labeled data, however, may now be rapidly shrinking thanks to new improvements, by Deep Mind, upon…