NLP News Cypher | 01.12.20
The Truth Hurts…
Perhaps, you can tell me what’s best.
I trained a GPT-2 medium model on 18K+ tweets.
Connected the model to Twitter’s API.
Every 30 minutes it says things - sometimes funny…
😭😭😭
Thinking of doing a blend including “interesting characters” on Twitter and retraining GPT-2 to have more relevant generation. For those interested in replicating, if you choose to use tweets for your training set, make sure your dataset doesn’t have tweets that reference media attachments like “Hey check this out: [VIDEO]”. If it does, some inferences will not be useful because you tend to generate tweets like this:
If you have cool ideas on who would be fun to add to the training set, hit me up on Twitter or comment below!
Side Note: Cool announcement mid-week. #cliffhanger 👀
Before we start, today’s column was inspired by the anthemic 1966 song “ Wild Thing” by The Troggs. Video Below 👇👇
Video starts with the band standing, instruments in hand, on what looks like a hallway of some kind. They then follow a femme fatale through a door into what looks like a back room but turns out they are actually in a middle of a subway station. 🤯🤯
Lead singer does rattle snake head movement the whole time. 🤟🤟
This Week:
GPT-2 for Tweeting [What you just read]
Neural Module Network Effects
Too Many Recaps, Not Enough Time
Lex’s Vision is 2020
Time for a Fireside Chat?
Reading Comprehension Evaluation Server
Using BERT for NLU
Dataset of the Week: AQuA
Neural Module Network Effects
Nitish Gupta et al. introduces a Neural Module Network model that is able to reason over a paragraph symbolically (arithmetic, sorting, counting) on numbers and dates. It also achieves SOTA on a subset of the DROP dataset.
According to source, code is dropping soon…
Paper:
Too Many Recaps, Not Enough Time
Every big tech company’s AI research arm has come out with a “Year in Review.” This past week it was Facebook and Google's turn.
My favorite blog post (from Facebook AI’s review) discussed the challenges of open-domain dialogue:
Facebook:
Google:
Me:
Lex’s Vision is 2020
I remember watching Lex’s 2019 video (seen here) and really enjoying it. Well, he has returned. And BTW, NLP gets a big shout-out. Transformers are kind of a big deal. Anyway, lucid recap of the current state of AI across NLP and Computer Vision.
Time for a Fireside Chat?
Wasn’t aware there was a compendium for this. But Microsoft Research shared a collage of various video interviews with the industry’s thought leaders.
Reading Comprehension Evaluation Server
They call it ORB (Open Reading Benchmark). You drop a single question answering model into ORB’s server and it evaluates on several reading comprehension datasets. When submitting your model, they require a docker image that will run on their VM with 4 vCPUs, 1 P100 GPU, and 26GB RAM for eval.
Using BERT for NLU
A fellow named Olivier Grisel fine-tuned BERT to convert an English user query into a representation for handling NLU on task-oriented dialogue. It was fine-tuned on SNIPS, a voice assistant dataset. The project was partly based on the Alibaba paper: https://arxiv.org/pdf/1902.10909.pdf.
Below is an example for intent classification/slots filling on a query:
>>> show_predictions("Book a table for two at Le Ritz for Friday night!",
... tokenizer, joint_model, intent_names, slot_names)____________________________________________________________________## Intent: BookRestaurant
## Slots:
Book : O
a : O
table : O
for : O
two : B-party_size_number
at : O
Le : B-restaurant_name
R : I-restaurant_name
##itz : I-restaurant_name
for : O
Friday : B-timeRange
night : I-timeRange
! : O
Notebook:
Colab:
Dataset of the Week: AQuA
We’re doing something new, from now on, we’ll highlight an NLP dataset every week.
Ok… back to AQuA… aka Algebra Question Answering with Rationales.
What is it:
“Algebraic word problem dataset, with multiple choice questions annotated with rationales.”
Sample:
"question": "A grocery sells a bag of ice for $1.25, and makes 20% profit. If it sells 500 bags of ice, how much total profit does it make?",
"options": ["A)125", "B)150", "C)225", "D)250", "E)275"],
"rationale": "Profit per bag = 1.25 * 0.20 = 0.25\nTotal profit = 500 * 0.25 = 125\nAnswer is A.",
"correct": "A"
Where is it?
Every Sunday we do a weekly round-up of NLP news and code drops from researchers around the world.
If you enjoyed this article, help us out and share with friends or social media!
For complete coverage, follow our twitter: @Quantum_Stat
