A hen’s-eye view of contemporary AI from NeurIPS 2019

By Alex Kolchinski, PhD Candidate at Stanford.

This yr, I had an opportunity to attend NeurIPS, essentially the most outstanding convention in synthetic intelligence and machine studying (AI/ML), to current a workshop paper. I’ve spent the previous couple of years engaged on a mixture of AI analysis in numerous subfields and tech startups and so have been following the evolution of AI with curiosity. This convention, bringing collectively because it does a few of the finest researchers and practitioners within the discipline, was a very good vantage level to gauge the state of and adjustments in how individuals are fascinated by and utilizing AI. Right here, I’ve collected a few of my impressions within the hopes that they is perhaps helpful to others. In the event you’re interested by different individuals’s views, Andrey Kurenkov collected some hyperlinks to various talks and key trends in his recent post, which can also be value a glance.

Essentially the most overarching theme I seen at NeurIPS was the maturation of deep studying as a set of methods. Since AlexNet gained the ImageNet problem resoundingly in 2012 by making use of deep studying to a contest beforehand dominated by classical pc imaginative and prescient, deep studying has attracted a really massive share of the eye throughout the discipline of AI/ML. Since then, the efforts of numerous researchers growing deep studying and making use of it to varied issues have completed issues like beating people at Go, coaching robotic fingers to resolve Rubik’s cubes, and transcribing speech with unprecedented accuracy. Successes like these have generated pleasure each throughout the AI neighborhood and elsewhere, with the mainstream impression tending in direction of an overestimate of what AI can really do, fueled by the extra narrowly circumscribed successes of latest, largely deep-learning powered, strategies. (Gary Marcus has an important latest essay speaking about this in additional element.)

Nonetheless, a perspective that I discover extra helpful than “the robots are coming” is the one I heard from Michael I. Jordan when he got here to Stanford to provide a chat during which he described trendy machine studying because the rising discipline of engineering which offers with knowledge. In keeping with this angle, I noticed various traces of inquiry at NeurIPS, that are growing the sector into extra nuanced instructions than “Got a prediction problem? Throw a deep net at it.” I’ll break down my impressions into three normal areas: making fashions extra sturdy and generalizable for the true world, making fashions extra environment friendly, and attention-grabbing and rising functions. Whereas I don’t declare that my impressions are a consultant pattern of the sector as a complete, I hope they’ll show helpful nonetheless.


Robustness and generalizability

One outstanding class of labor that I noticed at NeurIPS was that which addressed real-world necessities for efficiently deploying fashions different than simply excessive test-set accuracy. Whereas a canonical case of a profitable deep studying mannequin, like a picture classifier skilled on the ImageNet dataset, is profitable inside its personal area, the true world during which fashions have to be grounded and deployed is advanced and ambiguous in methods which fashions a lot deal with if they’re to be helpful in follow.

Considered one of these complexities is calibration: the flexibility of a mannequin to estimate the boldness with which it makes predictions. For a lot of real-world duties, it’s obligatory not solely to have an argmax prediction however to know the way seemingly that prediction is to be correct, in order to tell the load given to that prediction in subsequent decision-making. Numerous papers at NeurIPS addressed higher approaches to this complexity.

One other complexity is making certain that fashions are assigning applicable significance to options which are semantically significant and generalizable, which in a technique or one other contains illustration studying, interpretability and adversarial examples. A narrative I heard that illustrates the motivation for this line of analysis had its origins in a hospital, which had created a dataset of (if I bear in mind accurately) chest X-ray pictures with related labels of which sufferers had pneumonia and which didn’t. When researchers skilled a mannequin to foretell the pneumonia labels, its out-of-sample efficiency was wonderful. Nonetheless, additional digging revealed that in that hospital, sufferers prone to have pneumonia had been despatched to the “high-priority” X-ray machine, and lower-priority sufferers had been despatched to a different machine completely. It additionally emerged that the machines left attribute visible signatures on the scans they generated and that the mannequin had realized to make use of these signatures as the first characteristic for its predictions, resulting in predictions that weren’t based mostly on something semantically related to pneumonia standing and which might neither yield incremental helpful data within the unique hospital nor generalize in any option to different hospitals and machines.

This story is an instance of a “clever Hans” second, during which a mannequin “cheats” by discovering a quirk of the dataset it’s skilled on with out studying something significant and generalizable concerning the underlying process. I had an important dialog about this with Klaus-Robert Müller, whose paper on the phenomenon is nicely value a learn. I noticed various different papers at NeurIPS coping with interpretability of fashions, in addition to illustration studying, the associated examine of how fashions characterize knowledge. A notable subset of this work was in disentangled representations, an strategy that goals to induce fashions to be taught representations of knowledge which are composed of meaningfully or usefully factorized elements. An instance could be a generative mannequin of human faces that learns latent dimensions equivalent to hair coloration, emotion, and many others., thus permitting higher interpretability and management of the duty.

A ultimate path attracting a big quantity of consideration within the “what models learn” class was that of adversarial examples, that are knowledge factors which have semantically significant options corresponding to 1 class, however much less semantically significant options which bias a mannequin’s prediction in a special path – for instance, a photograph that appears like a panda bear to people however which accommodates noise that makes a mannequin predict it to be a tree. Latest work in adversarial coaching has made progress in making fashions extra resilient to such adversarial examples, and there have been various papers at NeurIPS on this vein. I additionally had a really attention-grabbing dialog with Dimitris Tsipras, who was a co-author on this paper, which discovered outcomes that counsel that picture classifiers could use some less-robust options for classification, which could be perturbed to generate adversarial examples with out modifying the extra sturdy options which people primarily deal with. That is an rising space of investigation and the literature is value a better look.

All in all, it seems that the neighborhood is spending appreciable effort in making fashions extra sturdy and generalizable to be used in the true world, and I’m excited to see what additional fruit this bears.



As the ability and applicability of deep studying grows, we’re seeing a transition of the sector from the Zero-to-1 section, during which a very powerful outcomes need to do with what’s or will not be doable in any respect, to a 1-to-n section, during which tuning and optimizing the methods beforehand discovered to be helpful turns into extra necessary. And simply because the deep studying revolution had its underlying roots within the higher availability of compute and knowledge, so too are essentially the most outstanding instructions on this space, which I noticed at NeurIPS involved with bettering the data-efficiency and the computational effectivity of fashions.

Information effectivity

Finally, deep studying relies on massive quantities of knowledge to be helpful, however amassing this knowledge and labeling it (for supervised approaches) are sometimes the costliest and troublesome levels of making use of deep studying to an issue. Numerous papers at NeurIPS needed to do with decreasing the severity of this subject. Many needed to do with self-supervised studying, during which a mannequin is skilled to characterize the underlying construction of a dataset through the use of implicit reasonably than specific labels, e.g., predicting pixels of a picture from neighboring pixels or predicting phrases in a textual content from adjoining phrases. One other strategy that various papers handled is semi-supervised studying, the place fashions are skilled on a mixture of labeled and unlabeled knowledge. And at last, weakly supervised studying has to do with studying fashions from imperfect labels, that are cheaper and simpler to gather than good or virtually good ones. Chris Ré’s group at Stanford, with their Snorkel challenge, are outstanding on this space and had not less than one paper on weakly supervised studying at NeurIPS this yr. This additionally falls below the “systems for ML” class talked about within the subsequent part.

One other outstanding path having to do with knowledge effectivity (and likewise linked to illustration studying) is that of meta/switch/multi-task studying. Every of those approaches seeks to have fashions effectively be taught representations that are helpful throughout duties, thereby rising the pace and data-efficiency with which new duties could be tackled, as much as and together with one- and even zero-shot studying (studying a brand new process from a single instance, or no examples in any respect). One attention-grabbing paper amongst many on these matters was this one, which introduces an strategy to buying and selling off regularization on cross-task vs. task-specific studying within the meta-learning setting.

One other path in knowledge effectivity, which I seen prominently at NeurIPS, needed to do with shaping the house inside which fashions be taught to higher mirror the construction of the world inside which they function. This may broadly be regarded as “stronger priors” (though it appears the time period “priors” itself is getting used much less often). Basically, by constraining studying with some prior data of how the world works, knowledge can be utilized for studying extra effectively inside this smaller house of prospects. On this vein, I noticed a few papers (here and here) bettering fashions’ skills to be taught representations of the 3D world by way of approaches knowledgeable by the geometric construction of the world. I additionally noticed a few papers (here and here, each from people at Stanford) which use pure language to floor their representations of what they be taught. That is an intriguing strategy as a result of we use pure language to floor and talk our notion of the world, and forcing fashions to be taught representations mediated by our languages in a way imposes real-world priors upon the fashions. A ultimate paper I’d point out within the class of priors in addition to this one, which confirmed surprisingly good efficiency on MNIST of networks “trained” by structure search alone – whereas this might not be instantly relevant, it’s suggestive of the diploma to which selecting community structure fastidiously (i.e., in a manner that displays the construction of a process) could make the training course of sooner and cheaper.

One ultimate path related to knowledge effectivity is that of privacy-aware studying. In some instances (and sure extra to return sooner or later), knowledge availability is bottlenecked by privateness constraints. Numerous papers I noticed, together with many within the space of federated studying, handled how one can be taught from massive quantities of knowledge with out compromising the privateness of the individuals or organizations from which the information originated.

Computational effectivity

In addition to knowledge effectivity, the effectivity with reference to computational assets – i.e., compute and reminiscence/storage – was additionally a outstanding path of many papers at NeurIPS. I noticed various papers having to do with the compression of fashions and embeddings (the illustration of the information utilized by fashions in sure settings). Shrinking fashions and embeddings/representations of knowledge reduces each computational and storage necessities, permitting extra “bang for the buck.” I additionally noticed some attention-grabbing work in biologically-inspired neural networks, reminiscent of this paper from Guru Raghavan at Caltech. One motivation on this space is that whereas there will likely be sure limits to what number of matrix multiplications and additions could be carried out per greenback/second on general-purpose hardware to push the capabilities of contemporary deep studying, it might be doable to make use of special-purpose hardware which extra carefully approximates the features of organic neurons to realize increased efficiency for sure duties. I heard a mixture of curiosity and skepticism round biologically-inspired approaches from fellow NeurIPS attendees: that is an space to observe for the 10+ yr horizon.


Instructions and functions

Lastly, whereas at NeurIPS I additionally discovered it very attention-grabbing to get a really feel for the higher-level traits in numerous subfields of AI/ML and a really feel for the completely different functions now doable, or turning into doable, due to latest advances in analysis. This part is extra of a smorgasbord than a story; skip round as curiosity dictates.

Graph neural networks

One space which I ought to point out seeing various papers round is that of graph neural networks. These networks are in a position to extra successfully characterize knowledge in settings with graph-like construction, however as I do know little or no about this path personally, I’ll as an alternative refer readers to the page of the NeurIPS workshop on graph illustration studying as a place to begin into the literature.

Reinforcement studying and contextual bandits

One other space during which I noticed a completely large quantity of labor was that of contextual bandits and reinforcement studying (RL). A number of approaches which I noticed various papers in had been hierarchical RL (associated to illustration studying) and imitation studying (in a way, setting priors for fashions by way of human demonstration). I additionally noticed various papers coping with long-horizon RL, consistent with latest success in RL duties requiring planning additional into the long run, e.g., the sport Montezuma’s Revenge. Numerous papers additionally needed to do with transferring from simulation to the true world (sim2real), together with OpenAI’s placing demonstration of educating a robotic hand to resolve a Rubik’s dice in the true world after coaching in-simulation. I additionally talked to Marvin Zhang from Berkeley about a paper he co-authored during which a robotic was skilled on movies of human demonstrations – “demonstration to real” reasonably than “simulation to real” studying.

Nonetheless, you will need to notice that in follow, RL for the true world, i.e., hardware/robotics, remains to be not fairly there. RL has discovered nice success in settings the place the state of the issue is totally representable in software program, like Atari video games or board video games like Go. Nonetheless, generalizing to the a lot messier actual world has proved tougher – even the OpenAI crew behind the Rubik’s dice challenge spent three months fixing the issue in-simulation after which virtually two years getting it to generalize to an actual robotic hand with an actual Rubik’s dice – and even then, with far lower than 100% reliability. It will likely be attention-grabbing to see how shortly new approaches to RL can sq. the circle of generalizing to the true world. I had an important dialog with Kirill Polzounov and Lee Redden from Blue River about this – they introduced a paper on a plugin they developed for OpenAI health club permitting individuals to shortly check RL algorithms on real-world hardware. I’m excited to see how shortly “RL for the real world” progresses – if we see an inflection level just like the one imaginative and prescient hit in 2012, the implications for robotics may very well be large.

Pure language processing

One other space value mentioning is NLP (pure language processing), during which I’ve performed some work personally. The Transformers/transferable language mannequin revolution remains to be bearing fruit, with various papers exhibiting good outcomes leveraging these methods. I used to be additionally intrigued by a paper that claimed unprecedented long-horizon efficiency for memory-augmented RNNs. It will likely be attention-grabbing to see if the pendulum swings again from “attention is all you need” again to extra conventional RNN approaches. It’s additionally value noting that NLP is beginning to hit its stride for real-world functions. I’ve just a few buddies and acquaintances engaged on startups within the discipline, together with Brian Li of Compos.ai, whom I bumped into at NeurIPS. I additionally loved peeking into the workshop on doc intelligence – it seems NLP for the authorized house is already a multi-billion greenback business! Broadly talking, pure language is the informational connective tissue of human society, and methods to use computational approaches to this buzzing net of data will solely develop sooner or later.


One other space I’ll deal with briefly, from private ignorance reasonably than unimportance, is that of SysML – i.e., programs for ML and ML for programs. That is an exploding discipline, as evidenced by the quite a few papers introduced at NeurIPS and the workshops within the discipline. One notably attention-grabbing speak was the one Jeff Dean gave on the ML for programs workshop – positively value a watch if you will discover a recording (please depart a remark in case you do). He and his crew at Google managed to coach a community to structure ASICs way more shortly than human engineers might and met and even surpassed the efficiency of ASICs laid out by people. Numerous different papers additionally confirmed compelling leads to optimizing all the pieces from reminiscence allocation to detecting faulty GPUs with the assistance of deep studying. Numerous papers additionally addressed the “systems for ML” path, such because the Snorkel paper talked about above.

Generative fashions

Generative fashions have reached a stage of great maturity and are actually getting used as a instrument for different instructions in addition to being a analysis path in their very own proper. The efficiency of the fashions themselves is now unimaginable, with fashions like BigGAN having beforehand established a photorealistic state-of-the-art for imaginative and prescient, and I noticed various papers yielding unbelievably good leads to conditional text-to-image technology, video-to-video mapping, audio technology, and extra. I’ve been fascinated by various downstream functions of those methods, together with some within the trend business and visible and musical inventive instruments, and I’m trying ahead to seeing what emerges within the business within the years to return. Functions of generative fashions in different fields of machine studying have additionally been attention-grabbing, together with fields like video compression – I talked to some people from Netflix about this, as it might show helpful for decreasing the bandwidth load on the Web from a video. (Netflix and Youtube alone use one thing like ⅔ of the bandwidth within the U.S.) Generative fashions are additionally being utilized in sim2real work in robotics, as beforehand talked about.



Lastly, for the sake of completeness, I’ll point out just a few extra areas that I witnessed smaller bits of. Autonomous driving remains to be seeing a big and heterogenous quantity of labor. Plainly we’re settling right into a state of incremental enchancment, the place each analysis and deployment of self-driving goes to occur in matches and begins over the subsequent a number of a long time (e.g., native meals supply with gradual, small automobiles and truck platooning are simpler issues than autonomous taxis in cities, and can seemingly see extra business progress sooner). However, deep studying for medical imaging seems to be maturing as a discipline, with quite a few refinements and functions nonetheless rising. Lastly, I used to be additionally intrigued by a paper in deep studying for mixed integer programming (MIP). Conventional “operations research” fashion optimization like that which could be framed as MIP issues drives large financial worth in business, and it will likely be attention-grabbing to see if deep studying proves to be helpful alongside older methods there as nicely.



Fashionable AI/ML, largely powered by deep studying, has exploded into a big and heterogeneous discipline. Whereas there’s a point of unsubstantiated hype about its prospects, there’s additionally loads of real worth to be derived from the progress of the final 7+ years, and plenty of promising instructions to be explored as the sector matures. I stay up for seeing what the subsequent decade brings, each in analysis and in industrial functions.

Due to Shengjia Zhao and Isaac Sheets for serving to edit this essay.

Original. Reposted with permission.


Bio: Whereas at Stanford, Alex Kolchinski labored on numerous subfields of AI/machine studying, initially with a deal with academic expertise, with analysis pursuits together with pure language processing and generative fashions. Earlier than Stanford, Alex was an Affiliate Product Supervisor at Google, a guide in knowledge science and product administration, and a software program engineer. Whereas attending faculty at UChicago, Alex earned a BS and MS in pc science (AI specialization) and a minor in statistics.


About the Author

Leave a Reply

Your email address will not be published. Required fields are marked *