The NLP Cypher | 11.21.21

PyTorch LIT (talkin’ bout Inference)

Model Size x 18 = Model Memory Required

A Convenient Collection of Simple Python Code Snippets

OpenAI’s API Goes Open Range

G5 Instances at AWS w/ A10G GPUs

Hop: Reading Files without Extracting Archive

InfraNodus | Text Analysis Software

Distributed Training w/ PyTorch Lightning and Ray

Papers to Read 📚

https://arxiv.org/pdf/2111.08609.pdf
https://arxiv.org/pdf/2111.07991.pdf
https://arxiv.org/pdf/2111.07935.pdf

Repo Cypher 👨‍💻

Improving DeBERTa using ELECTRA Style Pre-Training with Gradient-Disentangled Embedding Sharing. On GLUE it achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure.

A benchmark for Data Centric AI. It benchmarks how data modification can impact model’s performance. You can modify the training set and validation set, re-split the training set and validation set, or add data by non-crawler methods. The modification can be done by algorithms or programs or in combination with manual methods.

Dynamic-TinyBERT, a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss drop).

--

--

--

Subscribe to the NLP Cypher newsletter for the latest in NLP & ML code/research. 🤟

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Papers to Read on Spiking Neural Networks

How To Write A Statistical Learning Model In Excel To Predict Whether A Bank Note Is Fake Or Not

Time series forecasting with 2D convolutions

Supervised vs Unsupervised Learning

Day 43: 60 days of Data Science and Machine Learning Series

K Means Clustering for Imagery Analysis

Knowledge Distillation in a neural network

Survey of facial feature descriptors

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ricky Costa

Ricky Costa

Subscribe to the NLP Cypher newsletter for the latest in NLP & ML code/research. 🤟

More from Medium

The Dangers of Context-Insensitivity in NLP

A knowledge graph (NLP) based approach for the identification of factors influencing premature…

Challenges in using NLP for low-resource languages and how NeuralSpace solves them

Analyzing Scientific Documents with fine-tuned SciBERT NER Model and Neo4j