## 1. Project Overview & Quickstart (dair-ai/prompts-intro)
# prompts-intro
Open-source repository dair-ai/prompts-intro
### Repository Details
- **Repository:** [dair-ai/prompts-intro](https://github.com/dair-ai/prompts-intro)
- **Primary Language:** Code
*Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.*
## 2. Official Technical Reference & Guides (dair-ai/dair-ai.github.io)
## File: README.md
## DAIR.AI
[DAIR.AI](http://www.dair.ai) aims to democratize Artificial Intelligence (AI) research, education, and technologies.
Find out how you can start contributing to DAIR.AI in the [issues](https://github.com/dair-ai/dair-ai.github.io/issues) section. We thank all our contributors.
Join our [Discord](https://discord.gg/SKgkVT8BGJ)!
Feel free to reach out to me on [Twitter](https://twitter.com/omarsar0) for contributions, sponsorship, and collaborations.
---
## File: 404.md
---
layout: page
title: "Page Not Found"
excerpt: "Page not found. Your pixels are in another canvas."
sitemap: false
permalink: /404.html
---
Sorry, but the page you were trying to view does not exist --- perhaps you can try searching for it below.
---
## File: _config.yml
# Site wide configuration
title: DAIR.AI
locale: en_US
url: https://dair.ai
# Jekyll configuration
permalink: /:categories/:title/
markdown: kramdown
highlighter: rouge
sass:
sass_dir: _sass
style: compressed
plugins:
- jekyll-sitemap
- jemoji
kramdown:
auto_ids: true
footnote_nr: 1
entity_output: as_char
toc_levels: 1..6
enable_coderay: false
syntax_highlighter: rouge
# Site owner
owner:
name: DAIR.AI
avatar: dair-ai.png
bio: "Democratizing Artificial Intelligence Research, Education, and Technologies"
email: ellfae@gmail.com
disqus-shortname: dair-ai
twitter: dair_ai
facebook:
google:
plus: #OmarSar27
analytics: UA-158959084-1
verify:
ad-client:
ad-slot:
bing-verify:
github: dair-ai
stackoverflow: #http://stackoverflow.com/users/123456/username
linkedin: #username
xing: #username
instagram: #username
lastfm: #username
tumblr: #username
pinterest: #username
foursquare: #username
steam: #username
dribbble: #username
youtube: #username
soundcloud: #username
weibo: #username
flickr: #username
include: [".htaccess"]
exclude: ["lib", "config.rb", "Capfile", "config", "log", "Rakefile", "Rakefile.rb", "tmp", "less", "*.sublime-project", "*.sublime-workspace", "test", "spec", "Gruntfile.js", "package.json", "node_modules", "Gemfile", "Gemfile.lock", "LICENSE", "README.md"]
---
## File: _octopress.yml
# Default extensions
post_ext: md
page_ext: md
project_ext: md
# Found in _templates/
post_layout: post
page_layout: page
project_layout: project
# Format titles with titlecase?
titlecase: true
---
## File: CODE_OF_CONDUCT.md
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at ellfae@gmail.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq
---
## File: CONTRIBUTING.md
In the future we will be writing a more generic guidance on how to contribute to dair.ai. More details coming soon!
For the meantime, we have provided some guidance on how to contribute to our different projects in the different [issues](https://github.com/dair-ai/dair-ai.github.io/issues) and tasks available for contributions.
---
## File: index.md
---
layout: home
excerpt: "Democratizing Artificial Intelligence Research, Education, and Technologies"
tags: []
image:
feature: dair-header.png
credit: dair.ai
creditlink: https://www.dair.ai
---
---
## File: package.json
{
"name": "minimal-mistakes-theme",
"author": "Michael Rose ",
"homepage": "http://mmistakes.github.io/minima-mistakes/",
"repository": {
"type": "git",
"url": "git://github.com/mmistakes/minimal-mistakes.git"
},
"bugs": {
"url": "https://github.com/mmistakes/minimal-mistakes/issues"
},
"engines": {
"node": ">= 0.10.0"
},
"devDependencies": {
"grunt": "~0.4.1",
"grunt-contrib-clean": "~0.5.0",
"grunt-contrib-jshint": "~0.6.3",
"grunt-contrib-uglify": "~0.2.2",
"grunt-contrib-watch": "~0.5.2",
"grunt-recess": "~0.3.5",
"grunt-contrib-imagemin": "~0.2.0",
"grunt-svgmin": "~0.2.0"
}
}
---
## File: _data/authors.yml
# Authors
e-tony:
name: Tornike Tsereteli
web:
bio: "I am a M.Sc. Computational Linguistics student at University of Stuttgart. My research interests are Natural Language Understanding, Transfer Learning, and low-resource settings."
twitter:
instagram:
github: e-tony
avatar: tony.png
jadevaibhav:
name: Vaibhav Jade
avatar: dair-ai.png
harunuz:
name: Harun Uz
bio: "Software Developer at Loodos Tech and Gradute Student at Istanbul Technical University"
twitter: HrnUz
instagram: harunuuz
github: harunuz
avatar: harunuz.jpg
haneybarg:
name: Fernanda Guimarães
avatar: haneybarg.jpeg
rish-16:
name: Rishabh Anand
web: https://rish-16.github.io/
bio: "Rishabh is a Machine Learning research student currently using statistical analysis and modelling to study COVID-19 mutations as an intern at A*STAR, Singapore. He loves Natural Language Processing and is also a technical writer and active open-source contributor."
twitter: rishabh16_
github: rish-16
avatar: rishabh.png
dair.ai:
name: DAIR.AI
web: https://dair.ai/
bio: "Democratizing Artificial Intelligence Research, Education, and Technologies"
twitter: dair_ai
github: dair-ai
avatar: dair-ai.png
naraquev:
name: Nicolas Araque Volk
web: https://medium.com/@nicolas_araque
bio: "Hardware Knight. Artificial Intelligence Advocate and Data Driven Geek."
twitter: nicolas_araque
avatar: naraquev.png
lbourdois:
name: Loïck BOURDOIS
web: https://lbourdois.github.io/blog/
bio: "Data Scientist working at the Bordeaux Population Health Research Centre of INSERM University of Bordeaux."
avatar: lbourdois.png
KaiyuanGao:
name: kaiyuan
web: https://blog.csdn.net/Kaiyuan_stju
twitter: kaiyuan70895595
github: KaiyuanGao
avatar: kaiyuan.jpg
SufianDira:
name: Sufian Diraneyya
avatar: sufian.jpg
fclesio:
name: Flavio Clesio
web: https://flavioclesio.com
bio: "Machine Learning Engineer (NLP, CV, Marketplace RecSys)"
twitter: flavioclesio
instagram: flavioclesio
avatar: flavio.png
github: fclesio
VictorGarritano:
name: Victor Garritano Noronha
twitter: vic_garritano
github: VictorGarritano
avatar: victor.jpg
billy_rick:
name: Elvis Saravia
web: http://dair.ai
email: ellfae@gmail.com
bio: "Co-Founder at DAIR.AI"
avatar: white_background_elvis.png
twitter: omarsar0
instagram:
github: omarsar
google:
plus: #BillyRick
elvis:
name: Elvis Saravia
web: http://dair.ai
email: ellfae@gmail.com
bio: "Co-Founder at DAIR.AI"
avatar: white_background_elvis.png
twitter: omarsar0
instagram:
github: omarsar
google:
plus: #BillyRick
cfierro:
name: Constanza Fierro
email: constanza.fierro94@gmail.com
bio: "SWE at Youtube."
avatar: avatar_cfierro.png
github: cfierro94
twitter: constanzafierro
AntonioLopardo:
name: Antonio Lopardo
email: antonio.lopardo@outlook.com
bio: "MSc of computer science student at Politecnico di Milano. Data Science and NLP practitioner, writer at OpenMined."
avatar: antoniolopardo.jpeg
github: AntonioLopardo
twitter: AntonioLprd
viktor2k:
name: Viktor Karlsson
email: viktor2karlsson@gmail.com
bio: "Software Engineer and MSc of Machine Learning with a growing interest of NLP. Trying to stay on-top of recent developments within the ML field in general, and NLP in particular. Writing to learn!"
web: https://medium.com/@viktor2karlsson
avatar: viktor2k.png
github: Viktor2K
---
## File: _data/navigation.yml
# Site navigation links
- title: GitHub 🐙
url: https://github.com/dair-ai
- title: Blog ✍️
url: /posts/
- title: About ℹ️
url: /about/
- title: Discord 💬
url: https://discord.gg/SKgkVT8BGJ
- title: Meetups 🌐
url: https://www.meetup.com/dair-ai/
- title: Newsletter 📥
url: https://nlpnews.substack.com/
#- title: Contribute 👩💻
# url: https://forms.gle/HaGQyGmtB5Kkxvao9
---
## File: about/index.md
---
layout: page
title: About DAIR.AI
tags: [about, Jekyll, theme, responsive]
modified: 2014-08-08T20:53:07.573882-04:00
comments: false
image:
feature: about.jpg
credit: iBelmopan
creditlink: http://www.ibelmopan.com/
---
DAIR.AI aims to democratize Artificial Intelligence (AI) research, education, and technologies.
\\
We provide professional and consulting services for AI companies, including strategic guidance, professional training, and technical consulting services.
\\
Reach out to hello@dair.ai for business inquiries.
\\
All our projects are hosted on [GitHub](https://github.com/dair-ai).
\\
Follow our Twitter page [here](https://twitter.com/dair_ai).
---
## File: current/index.md
---
layout: page
title: Research and Projects
tags: [about, Jekyll, theme, responsive]
modified: 2014-08-08T20:53:07.573882-04:00
comments: false
image:
feature: about.jpg
credit: iBelmopan
creditlink: http://www.ibelmopan.com/
---
### Mental Disorder Detection
This study relates to the detection and analysis of mental disorders in the realm of social media, in particular Twitter. We are interested in extracting linguistic, emotional and behavioral styles for the task of mental disorder classification. We leverage crowd-sourcing concepts and propose a novel data collection mechanism for obtaining reliable online patient datasets.
* [:bar_chart: Current Demonstration](http://bit.ly/ideamidas)
### (EmoViz) - Emotion Analysis / Visualization for Intention Identification
This work focuses on utilizing emotional knowledge to further investigate user behavior, in particular user intention or user interest identification. Our understanding is that emotions have a tremendous role in how we perceive and interact with the world around us. To that end, we adopt emotion detection algorithms to understand how users are feeling about a product for a given period of time. Through such emotional analysis, we are able to identify trending products as shown in the demonstration provided below.
* [:bar_chart: Demonstration](http://bit.ly/emoviz)
* [:page_facing_up: Paper](http://bit.ly/1OnoC1S)
### Multilingual Emotion Detection / Classification
Emotion classification algorithms deal with the problem of predicting the emotion portrayed by a microblog post. Our in-house algorithms are able to achieve remarkable performances, even when applied to different domains and languages. Currently, we support emotion classification for English, Spanish and French. In future work, we are seeking to extend to other complex languages such as Indonesian, Japanese and Chinese.
* [:smile: Multilingual Emotion Classifier - Demo](http://bit.ly/ilmeda)
* [:page_facing_up: Paper](http://bit.ly/1sjzq7T)
### Event Detection
This field of study is related to the detection of trending events on social media platforms. Platforms such as Twitter contain many events happening around the world but it's important to recommend the more relevant and important ones. Adopting graph-based algorithms, we are able to identify events and obtain significant results as compared to techniques that rely on popular algorithms such as LDA.
* [:page_facing_up: Paper](http://bit.ly/1TBTiJ)
### Depression Support System
By combining studies related to interest identification, mental disorders, event detection and emotion detection, we are able to design a complex but efficient depression support system. Such system acts as a recommendation platform for depressed social network users. By understanding user's emotions, we can obtain their current interests and then suggest content through events or concept extraction. More to come... :construction_worker:
### Interest Identification
These set of studies aim at building dynamic algorithms that are able to extract online users' interests. Currently, we rely on graph-based contextual approaches for interest identification, but are looking into other areas related to behavior analysis such as emotion analysis. By combining these two kind of approaches, we are now able to build more reliable and efficient algorithms for interest identification from different platforms. :construction_worker:
### Reviews Helpfulness through Opinion Mining
This study relates to the use of state-of-the-art Opinion Mining techniques to deduce how helpful reviews can be. Our approach focuses on the emotional styles used by online users, particularily those users talking about different features of products and services. Once the emotional styles are obtained, we employ clustering and classification techniques to further understand the consumers' position through the content found in their reviews. Our approaches also works well with reviews submitted to different domains (e.g., movies and books). :construction_worker:
### Secure URL Shortner
A url shortner is a service that basically renders a shorter url for the one being provided (usually long). In this project, we implemented a service in a secure way that allows users to auto-generate a short URL that is secure and shareable among users with proper permission. Additionally, authors and users with the proper permission will be able to access a dashboard and obtain insights into the traffic information pertaining to the URL provided. Our hope is to provide a safe service using state-of-the-art security tools. [Code](https://github.com/wisebits/url-shortner)
---
## File: posts/index.md
---
layout: post-index
title: All Posts
excerpt: "A List of Posts"
---
---
## File: posts/_posts/2018-08-11-deep-learning-for-nlp-pytorch-tensorflow.md
---
layout: post
title: "Deep Learning for NLP: PyTorch vs Tensorflow – PyCon Taiwan 2018"
author: billy_rick
modified:
comments: true
excerpt: "Deep Learning for NLP: PyTorch vs Tensorflow – PyCon Taiwan 2018"
tags: []
---
In this talk, I discuss some of the best practices and latest trends in natural language processing (NLP) research. The main goal is to provide a comprehensive comparison between machine learning frameworks (PyTorch and Tensorflow) when used for NLP-related tasks, such as sentiment analysis and emotion recognition from textual data. I will cover how to program and train widely-used algorithms, such as neural word embeddings and long short-term memory (LSTM) networks, for sentence classification. I will discuss some challenges and opportunities in deep learning for NLP research together with the advantages and disadvantages of using PyTorch and Tensorflow.
{% include video id="xteaK6kSYnA" provider="youtube" %}
\\
[Slides](https://docs.google.com/presentation/d/1cf2H1qMvP1rdKUF5000ifOIRv1_b0bvj0ZTVL7-RaVE/edit?usp=sharing)
---
## File: posts/_posts/2018-10-19-deep-learning-emotion-recognition-pytorch-tensorflow.md
---
layout: post
title: "Deep Learning Based Emotion Recognition with PyTorch and TensorFlow"
author: billy_rick
modified:
comments: true
excerpt: "Deep Learning Based Emotion Recognition with PyTorch and TensorFlow"
tags: []
---
In these tutorials we are going to learn how to train deep neural networks, such as recurrent neural networks (RNNs), for addressing a natural language task known as emotion recognition. We will cover everything you need to know to get started with NLP using deep learning frameworks such as TensorFlow and PyTorch. We will cover the common best practices, functionalities, and steps you need to understand the basics of TensorFlow’s and PyTorch’s APIs to build powerful predictive models via the computation graph. In the process of building our models, we will compare PyTorch and TensorFlow to let the learner appreciate the strengths of each tool.
\\
Below is a clip demonstrating what the notebooks contains:
\\
\\
TensorFlow version ([Colab Notebook](https://colab.research.google.com/drive/14I-31WuynLg1B0RQHWwwRgBmqTlDgkwV) | [GitHub](https://github.com/dair-ai/emotion_recognition_tensorflow))
\\
PyTorch version ([Colab Notebook](https://colab.research.google.com/drive/1p51HFDExl7XagWfSQE5leDQwBC6I_e3D) | [GitHub](https://github.com/dair-ai/emotion_recognition_pytorch))
---
## File: posts/_posts/2018-10-24-nlp-overview.md
---
layout: post
title: "Modern Deep Learning Techniques Applied to Natural Language Processing"
author: billy_rick
modified:
comments: true
excerpt: "Modern Deep Learning Techniques Applied to Natural Language Processing"
tags: []
image:
thumb: nlp_home.png
---
Do you want to learn more about the modern deep learning techniques applied in NLP, and keep track of emerging trends, dataset/code releases, and state of the results in the field? In collaboration with [Soujanya Poria](https://twitter.com/soujanyaporia) (an NLP researcher), we are open sourcing an online learning resource for NLP researchers and developers called [“NLP Overview”](https://github.com/omarsar/nlp_overview).
\\
The project contains an overview of recent trends in deep learning based natural language processing (NLP). It summarizes implementation details behind deep learning techniques such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), and reinforcement learning used for various NLP tasks and applications. The site also contains a summary of state of the art results for NLP tasks such as machine translation, question answering, and dialogue systems. You can find the learning resource at the following address: [https://nlpoverview.com](https://nlpoverview.com/). A snapshot of the website is provided below:
\\
\\
*The main motivations for this project are as follows:*
- Maintain an up-to-date learning resource that integrates important information related to NLP research, such as state of the art results, theoretical details, emerging concepts and applications, new benchmark datasets, code/dataset releases, etc.
- Create a friendly and open resource to help guide researchers and anyone interested to learn about modern techniques applied to NLP
- A collaborative project where expert researchers can suggest changes (e.g., incorporate SOTA results) based on their recent findings and experimental results
\\
This book is published as open source and compliments well with other popular NLP resources such as Sebastian Ruder’s NLP Progress website. We are calling for contributions from the NLP and machine learning community to help us improve and keep the site up-to-date. We have created a [GitHub repository](https://github.com/omarsar/nlp_overview) for the website where pull requests can be made.
---
## File: posts/_posts/2018-12-22-bias-in-sentiment-analysis.md
---
layout: post
title: "Examining Gender and Race Bias in Sentiment Analysis Systems"
author: billy_rick
modified:
comments: true
excerpt: "Examining Gender and Race Bias in Sentiment Analysis Systems"
tags: []
image:
thumb: nlp_home.png
---
\\
A pressing issue in machine learning (ML) research is to examine whether automated algorithms discriminate against gender or race when making decisions. In a [recent study](http://aclweb.org/anthology/S18-2005), Kiritchenko and Mohammad conducted an extensive analysis to determine whether hundreds of sentiment analysis systems accentuate inappropriate human biases and how these are manifested.
## Motivation
Automatic systems are beneficial to society but as they improve in predictive performance, comparable to human capabilities, they could perpetuate inappropriate human biases. In the context of sentiment analysis (SA), these biases may come in many forms. For instance, an SA system may consider the messages conveyed by a specific gender or race to be less positive simply because of their race and gender. Therefore, even when it is not clear how biases are manifested, it is the duty and responsibility of the natural language processing (NLP) system developer to proactively address these issues.
## Contributions
In NLP, the sources of the bias may derive from the training data or language resources such as lexicons and word embeddings that a machine learning algorithm is designed to leverage for building the predictive model.
\\
There have been many works in computer vision and NLP that aim to detect these inappropriate biases, but few have focused on a large set of systems that aim to solve the same task, which is very common in the field of machine learning.
\\
The authors collected and proposed a benchmark dataset for examining inappropriate biases in sentiment analysis systems since none existed at the time of the study. Then 219 automatic sentiment analysis systems were examined for inappropriate biases using the dataset. The goal of the research was simply to examine whether the sentiment analysis systems were consistently assigning higher or lower sentiment intensity scores to sentence pairs involving a particular race or gender.
## Dataset
A benchmark dataset, coined as Equity Evaluation Corpus (EEC), was compiled to test the fairness of the sentiment analysis systems. Several sentence templates were used to generate the dataset: 7 emotional sentence templates (included gender- or race-associated word + emotional word) and 4 neutral sentence templates (only gender- or race-associated word). (See the 11 templates in the table below)
\\
The `` placeholder was instantiated by African American or European American names such as ‘Latisha’ and ‘Ellen’ and noun phrases such as ‘my daughter’ and ‘my son’. Whereas, the `` placeholder included an emotion state word with varying intensities or emotional situation word. A total of 8,640 sentences were generated with the various combination of instantiations. (Refer to the [paper](http://aclweb.org/anthology/S18-2005) for the full list of noun phrases, names, and emotion words). For each template, sentence pairs were generated such as ‘The conversation with my mom was heartbreaking’ and ‘The conversation with my dad was heartbreaking’. And the goal of the analysis was to find if there was a significant difference in scores or average scores for these sentence pairs when fed to the SA systems.
## The Sentiment Analysis Systems
The 219 automatic sentiment analysis systems that participated in the [SemEval-2018 Task 1: Affect in Tweets](https://competitions.codalab.org/competitions/17751) were individually evaluated for discrimination based on gender and race bias. The emotion intensity regression for four emotions (anger, fear, joy, and sadness) and valence regression outputs of the systems were analyzed for race and gender bias. The outputs were scores between 0 and 1. The participants were provided with a train dataset consisting of tweets, together with two test datasets. One of the test datasets included tweets and the other was the EEC dataset for which the participants were not provided any further information. Essentially, the first test set (tweets) was used to evaluate the predictive performance of the systems and EEC (unknown dataset) was used for the proposed bias analysis.
\\
Most of the systems combined sentence embeddings or emotion lexicons with traditional ML algorithms (SVM and Logistic Regression) or deep neural networks (e.g., LSTMs and Bi-LSTMs). The embeddings were obtained either through a [distant supervision](https://web.stanford.edu/~jurafsky/mintz.pdf) corpus, pre-trained models, or manually trained embeddings. The lexicons were often derived from the [NRC](https://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm) emotion and sentiment lexicons. A baseline SVM system was also trained with word unigrams as features. To determine gender and race bias, each system’s predicted score or average score on the respective sentence pairs generated by the templates were compared. Very high differences in scores indicated a higher bias by the systems.
## Gender Bias Results
The study reports that on the four emotion intensity prediction tasks a whopping 75% to 86% of the systems consistently scored sentences of one gender higher than another, even when all that changed was the noun phrases or names in the sentence pairs. This phenomenon was more evident in the emotions of anger and joy. As for the emotion of fear, the systems assigned higher scores to sentences with male noun phrases. Sadness was more balanced than the rest of the emotions. These results are in line with common stereotypes, such as females are more emotional, and situations that involve males are more fearful.
\\
Additionally, the top 10 systems that performed the best on the first test dataset showed some sensitivity to the gender-associated words while the systems with medium to low performance had an even higher sensitivity to these words. Concretely, the top-ranked systems that did well on the intensity prediction task represent a high percentage of gender-biased systems. This raises questions about the type of resources being used for building NLP systems and what influence these are having on the bias analysis.
## Race Bias Results
The race bias analysis produced similar results as compared to the gender bias analysis. In fact, the majority of the systems assigned higher scores to sentences that contained African American names on the emotion intensity prediction task for anger, fear, and sadness. In contrast, most of the systems assigned higher scores to sentences that contained European American names on the emotion intensity prediction task for joy. These results reflect the stereotypes found in a [previous study](https://link.springer.com/article/10.1023/A:1022986429748) which report that African Americans are commonly associated with more negative emotions.
## The Future
As future work, the authors will continue to investigate the systems (and its components) to locate the exact source/s of the identified biases. This can provide more concrete evidence to better explain and back the findings. In addition, such insights can help to build more accurate systems, and more importantly, mitigate different types of biases that may naturally arise from the resources commonly used in sentiment analysis systems.
\\
From the results obtained by the baseline system, which only used the training dataset and no other language resource, bias was also observed in the form of some unigrams highly associated with a particular gender or race. Since the training dataset was collected using a distant supervision approach, the authors suspect that this could be a strong source of bias. Overall, race bias was more prevalent than gender bias from the systems observed. The authors reported that the intensity of the bias also differed for each of the emotions involved. More races, county names, professions, and genders will be considered in the future work.
## Requests for Research
- The EEC corpus only presents one setting to examine the fairness of sentiment analysis systems. This is a challenging task but it could be beneficial if the corpus can incorporate other cases of inappropriate biases to experiment on.
- Crowdsourcing efforts could help to improve the quality of the corpus. In addition, you can also try other types of systems besides SA. How about emotion recognition systems or emoji recognition systems?
- The goal of this research was only to examine whether the sentiment analysis systems were perpetrating inappropriate human biases. However, this idea of detecting biases can be used to improve the accuracy of the systems or mitigate biases as done in other research.
- I believe that including context (e.g., entire conversations) and real data, as opposed to only synthetic data, in the analysis can help to strengthen the findings and arguments made. In fact, more interesting insights may emerge from the datasets.
*Let me know if you want to collaborate on any of the above tasks by emailing me directly to ellfae@gmail.com.* This was originally posted on my publication ([dair.ai](https://medium.com/dair-ai/examining-gender-and-race-bias-in-sentiment-analysis-systems-b04b269a653)), where you can find further discussions.
---
## File: posts/_posts/2019-01-10-emnlp-emotion-recognition.md
---
layout: post
title: "EMNLP 2018 Oral Presentation on Affective Computing (Emotion Recognition)"
author: billy_rick
modified:
excerpt: "NLP Newsletter"
comments: true
excerpt: "EMNLP 2018 Oral Presentation on Affective Computing (Emotion Recognition)"
tags: []
image:
thumb: emnlp2018-emotion-recognition.png
---
\\
This is an oral presentation I delivered in Brussels, Belgium at the EMNLP 2018 conference for the paper ["CARER: Contextualized Affect Representations for Emotion Recognition"](https://aclanthology.coli.uni-saarland.de/papers/D18-1404/d18-1404)
## Abstract
\\
Emotions are expressed in nuanced ways, which varies by collective or individual experiences, knowledge, and beliefs. Therefore, to understand emotion, as conveyed through text, a robust mechanism capable of capturing and modeling different linguistic nuances and phenomena is needed. We propose a semi-supervised, graph-based algorithm to produce rich structural descriptors which serve as the building blocks for constructing contextualized affect representations from text. The pattern-based representations are further enriched with word embeddings and evaluated through several emotion recognition tasks. Our experimental results demonstrate that the proposed method outperforms state-of-the-art techniques on emotion recognition tasks.
\\
[Link to Video](https://vimeo.com/306129121)
\\
[Link to Paper](https://aclanthology.coli.uni-saarland.de/papers/D18-1404/d18-1404)
---
## File: posts/_posts/2019-01-10-nlp-highlights-2018.md
---
layout: post
title: "NLP 2018 Highlights (Free 70+ Pages PDF Report)"
author: billy_rick
modified:
comments: true
excerpt: "NLP 2018 Highlights"
tags: []
image:
thumb: 2018-nlp-highlights.png
---
\\
It has been a big year for the field of natural language processing (NLP) and for machine learning as a whole. There have been many trends and breaking stories, with state-of-the-art results and new interesting research directions emerging. We owe all of this progress to the brilliant researchers around the world and the millions of developers devoting their full time to improve tools that make it easier for everyone to learn and progress the field. We have witnessed a rise in transfer learning and other niche areas of research such as AI Ethics.
\\
In this post, I am announcing the official release of a report called “NLP 2018 Highlights”. (Find the downloadable PDF at the end of this post).
\\
In this report, I have provided a summary of all the biggest NLP stories of the year (2018) coming from both academia and the industry. I hope this report serves as a guide for the researcher and developer that wishes to start learning about this field and also for the expert who wishes to brush up on some of the latest advancements. The selected topics are based purely on personal observation, so it is highly possible that I missed other important stories. In fact, the stories shared here are an extended compilation of the NLP Newsletter published weekly on the [dair.ai](https://medium.com/dair-ai) publication. I have made an effort to categorize the stories in the best possible way so that the report could benefit as many readers as possible. Please note that the report is meant to be non-technical for the purpose of reaching as many diverse readers as possible. The stories are mainly categorized by the following key topics: AI ethics, research publications, trends, education, resources, industry, and much more.
\\
Below is a preview of the table of contents inside the report:
\\
\\
And finally...
\\
[PDF of “NLP 2018 Highlights”](https://www.dropbox.com/s/ez3l47l7mg8zjbf/NLP_2018_Highlights.pdf?dl=0)
---
## File: posts/_posts/2020-01-12-NLP_Newsletter[PT-BR]_Tokenizers,_TensorFlow_2_1,_TextVe.md
---
layout: post
title: "NLP Newsletter #1 [PT-BR]: Tokenizadores, TensorFlow 2.1, Vetorização de Texto, TorchIO, Déficits de NLP,…"
author: billy_rick
modified:
comments: true
excerpt: ""
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_1.png
---
Olá e Feliz Ano Novo! Devido a vários pedidos, eu decidi trazer de volta a **Newsletter de NLP**. Dessa vez irei mantê-la pequena e focada (também mantida no [repositório](https://github.com/dair-ai/nlp_newsletter)). O objetivo dessa newsletter é te manter informado em alguns dos mais recentes e interessantes tópicos relacionados a NLP e ML (em algumas categorias) sem tomar muito tempo do seu dia ocupado.
# Publicações 📙
***Sistema de IA para ressonância de câncer de mama***
A DeepMind publicou um novo artigo na Nature intitulado “[International evaluation of an AI system for breast cancer screening](https://www.nature.com/articles/s41586-019-1799-6)” (Avaliação internacional de um sistema de IA para ressonância de câncer de mama). De acordo com os autores, o trabalho é sobre a avaliação de um sistema de IA que ultrapassa especialistas humanos na ressonância de câncer de mama. Se isso é realmente alcançável pelos sistemas de IA atuais ainda está em debate e há uma crítica contínua nesse tipo de sistema e como ele é avaliado. Aqui está um [pequeno resumo](https://www.nature.com/articles/d41586-019-03822-8) do artigo.
***Extração de Informação***
O Pankaj Gupta liberou publicamento sua tese de Ph.D. intitulada “[Neural Information Extraction From Natural Language Text](https://www.researchgate.net/publication/336739252_PhD_Thesis_Neural_Information_Extraction_From_Natural_Language_Text)” (Extração neural de informação de texto de linguagem natural). A discussão principal é como extrair eficientemente as relações semânticas de texto em linguagem natural usando abordagens neurais. Tal esforço em pesquisa visa contribuir com a construção estruturada de bases de conhecimento, que possam ser usadas em uma série de aplicações de NLP como busca na web, respostas de questões, dentre outras.
***Recomendações Melhoradas***
Pesquisadores no MIT e na IBM desenvolveram um [método](http://news.mit.edu/2019/finding-good-read-among-billions-of-choices-1220) (publicado na NeurIPS ano passado) para categorização, surgimento, e busca de documentos relevantes baseados numa combinação de três ferramentas altamente usadas para análise de texto: *modelagem de tópicos*, *embeddings de palavras*, e *transporte ideal*. O método também dá resultados promissores para a classificação de documentos. Tais métodos são aplicáveis em uma grande variedade de cenários que requerem sugestões melhoradas e mais rápidas em grande escala tal como sistemas de busca e recomendação.
# ML e NLP Criatividade e Sociedade 🎨
***Carreiras em IA***
O [relatório](https://hai.stanford.edu/sites/g/files/sbiybj10986/f/ai_index_2019_report.pdf) do índice de IA de 2019 sugere que há mais demanda que suprimento de praticantes de IA. Entretanto, há vários aspectos de empregos relacionados a IA como transições de carreira e entrevistas que ainda não são propriamente definidas.
Nesse [post](https://towardsdatascience.com/how-i-found-my-current-job-3fb22e511a1f), Vladimir Iglovivok detalha sua carreira e aventura em ML desde a construção de sistemas de recomendação tradicionais a construção de modelos espetaculares de visão computacional que ganharam competições no Kaggle. Ele agora trabalha em veículos autônomos na Lyft, mas a [jornada](https://towardsdatascience.com/how-i-found-my-current-job-3fb22e511a1f) de chegar lá não foi tão fácil.
Se você está realmente interessado e sério em uma carreira em IA, a companhia do Andrew Ng, deeplearning.ai, fundou Workera, que visa a especificamente ajudar cientistas de dados e engenheiros de aprendizado de máquina com suas carreiras em IA. Obtenha o relatório oficial [aqui](https://workera.ai/candidates/report).
# Ferramentas e Base de Dados de ML/NLP ⚙️
***Um tokenizador ultrarrápido***
Hugging Face, a startup de NLP por trás dos Transformers, liberou de forma open-sourced os Tokenizers, uma implementação ultrarrápida de tokenização que pode ser usada em pipelines modernos de NLP. Cheque o [repositório no GitHub ](https://github.com/huggingface/tokenizers) para a documentação em como usar os Tokenizers.
*Tokenizers — Python*
O TensorFlow 2.1 incorpora uma nova camada de [Vetorização de Texto](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization) que permite que você lide facilmente com strings brutas e execute eficientemente uma normalização de texto, tokenização, geração de n-gramas, e indexação de vocabulário. Leia o lançamento aqui e cheque o [notebook Colab do Chollet](https://colab.research.google.com/drive/1RvCnR7h0_l4Ekn5vINWToI9TNJdpUZB3) demonstrando como usar a feature para classificação de texto fim-a-fim.
***NLP e ML para Busca***
Um dos campos que fez tremendo progresso no ano passado foi NLP com uma gama de melhoras e direções novas de pesquisa. Um dos domínio que pode potencialmente se beneficiar de transfer learning em NLP é a *busca*.
Apesar de a busca pertencer ao campo de recuperação da informação, há uma oportunidade para construir engines que melhores a busca semântica usando técnicas modernas de NLP como representações contextualizadas de um modelo baseado em transformer como [BERT](https://arxiv.org/abs/1810.04805). O Google liberou um [post de blog](https://www.blog.google/products/search/search-language-understanding-bert/) alguns meses atrás discutindo como eles estão alavancando modelos BERT para melhorar e entender buscas.
Se você está curioso sobre como representações contextualizadas podem ser aplicadas à busca usando tecnologias abertas tais como Elasticsearch e Tensorflow, você pode dar uma olhada ou nesse [post](https://towardsdatascience.com/elasticsearch-meets-bert-building-search-engine-with-elasticsearch-and-bert-9e74bf5b4cf2) ou [aqui](https://towardsdatascience.com/building-a-search-engine-with-bert-and-tensorflow-c6fdc0186c8a).
***Análise médica de imagem***
[TorchIO](https://github.com/fepegar/torchio) é um pacote de Python baseado na biblioteca popular chama PyTorch. TorchIO oferece funcionalidades para fácil e eficientemente ler e amostrar imagems médicas 3D. Características incluem transformações espaciais para aumento e pré-processamento de dados.
[fonte](https://github.com/fepegar/torchio)
# Ética em IA 🚨
***Comportamento fraudulento na comunidade de ML***
Isso acabou de sair! O primeiro lugar de uma competição do Kaggle foi desqualificado por uma atividade fraudulenta. O time usou uma tática esperta mas irresponsável e inaceitável ao vencer o primeiro lugar da competição. Aqui está a [história completa](https://www.kaggle.com/c/petfinder-adoption-prediction/discussion/125436). Essa história realça um dos muitos dos comportamentos inaceitávies que a comunidade de aprendizado de máquina quer mitigar. O uso apropriado e ético de tecnologias de ML é o único caminho a seguir.
***Viés de gênero em tradução de máquina***
A respeito do tópico se tradução de máquina reflete viés de gênero, um grupo de pesquisadores publicou esse excelente [artigo](https://arxiv.org/abs/1809.02208) apresentando um estudo de caso usando o tradutor do Google. Uma das descobertas dos autores afirma que o tradutor do Google “exibe uma tedência forte a padrões masculinos, em particular para campos ligados a distribuições desbalanceadas de gênero tal como trabalhos STEM.”
***Viés e Justiça em ML***
Se você quer se atualizar com tudo de ética e justiça em IA, esse é um ótimo [episódio de podcast](https://twimlai.com/twiml-talk-336-trends-in-fairness-and-ai-ethics-with-timnit-gebru/) com a participação de Timnit Gebru e apresentado por TWIML.
Timit é uma pesquisadora proeminente em justiça em ML que, junto com Eun Seo Jo, publicou um [artigo](https://arxiv.org/abs/1912.10389) onde identificam cinco abordagens chave em práticas de coleta de documentos em arquivos que podem prover métodos mais confiáveis para coleta de dados em ML sociocultural. Isso pode potencialmente levar a um método de coleta de dados mais sistemático, ganhado de uma pesquisa colaborativa.
Sina Fazelpour e Zachary Lipton recentemente publicaram um [artigo](http://zacklipton.com/media/papers/fairness-non-ideal-fazelpour-lipton-2020.pdf) onde argumentam que, devido à natureza de como nosso mundo não ideal surgiu, é possível que ML justo, baseado no pensamento ideal, pode potencialmente levar a políticas e intervenções mal guiadas. Na verdade, suas análises demonstraram “que deficiências em algoritmos de ML refletem problemas maiores encarados pela abordagem ideal.”
# Artigos e posts de blogs ✍️
***Déficits em NLP***
Benjamin Heinzerling publicou um artigo interessante no The Gradient onde eles discute áreas onde NLP falha, como compreensão de argumentos e raciocínio de senso comum. Benjammin faz referência ao [artigo](https://www.aclweb.org/anthology/P19-1459/) recente por Nivin & Kao, que desafia e questiona as capacidades de transfer learning e modelos de lingauem para entendimento de linguagem natural (NLU) em alto nível. [Leia](https://thegradient.pub/nlps-clever-hans-moment-has-arrived/) mais sobre esse excelente resumo da análise executada na pesquisa.
***Destaques de NLP e ML em 2019***
Para o ano novo, eu liberei um [relatório](https://medium.com/dair-ai/nlp-year-in-review-2019-fb8d523bcb19) documentando alguns dos destaques mais interessantes em NLP e ML pelos quais passei em 2019.
Sebastian Ruder também escreveu recentemente um [post de blog](https://ruder.io/research-highlights-2019/) excelente e detalhado sobre as top dez direções de pesquisa em ML e NLP que ele achou impactante em 2019. Na lista estão tópicos como pré treino não supervisionado e universal, ML e NLP aplicados à ciência, aumentando modelos pré treinados, Transformers eficientes e de longo alcance, entre outros.
*“VideoBERT (*[*Sun et al., 2019*](https://arxiv.org/abs/1904.01766)*), uma variante recente multimodal do BERT que gera “tokens” de vídeos dado uma receita (acima) e prediz os tokens futuros em escalas de tempo diferentes dado um token de vídeo(abaixo).” —* [*fonte*](https://arxiv.org/pdf/1904.01766.pdf)
A Google AI de pesquisa publicou um [resumo](https://ai.googleblog.com/2020/01/google-research-looking-back-at-2019.html) da pesquisa conduzida por eles durantes o ano, e as direções futuras de pesquisa que eles estão prestando atenção.
# Educação em ML/NLP 🎓
***Democratizando educação em IA***
Num esforço de democratizar educação em IA e para educar as massas sobre as implicações de tecnologia de IA, a Universidade de Helsinki fez uma parceria com a Reaktor para lançar um curso brilhante (de graça) que cobre os fundamentos de IA. O [curso popular](https://www.elementsofai.com/) se chama “Elementos de AI” e inclui tópicos tal como ética em IA, filosofia em IA, redes neurais, regra de Naive Bayes, dentre outros tópicos.
Stanford CS224N está de volta com outra [iteração](http://web.stanford.edu/class/cs224n/) do popular curso de “Natural Language Processing with Deep Learning”. O curso oficialmente começou em 7 de Janeiro dessa ano, então se você quiser acompanhar, vai para o site para o programa completo, slides, vídeos, sugestões de leitura de paper, etc.
***Top livros de NLP e ML***
Eu tweetei meus top livros de ML e NLP práticos e teóricos, e foi bem recebido. Eu gostaria de compartilhar aquela lista aqui via tweet: https://twitter.com/omarsar0/status/1214547402838986754?s=20
***Machine Learning com Métodos de Kernel***
Métodos de Kernel tais como PCA e k-means estão por aqui há muito tempo, e isso é porque eles têm sido aplicados com sucesso a uma variedade grande de aplicações tal quais grafos e sequências biológicas. Cheque esse conjunto compreensível de [slides](http://members.cbio.mines-paristech.fr/~jvert/svn/kernelcourse/slides/master2017/master2017.pdf) cobrindo um alcance grade de métodos de kernel e suas aplicações internas. Aqui está também um ótimo [blog](https://francisbach.com/cursed-kernels/) (mantido por Francis Bach) discutindo aspectos de métodos de kernel e outros tópicos de aprendizado de máquina.
# Menções Honrosas ⭐️
Aqui está uma lista de histórias dignas de nota e da sua atenção:
- John Langford cuida desse incrível [blog](https://hunch.net/) que discute teoria de aprendizado de máquina
- Muitas das tecnologias da indústria orientadas a ML têm usado máquinas de Gradient Boosting por anos. Cheque esse [post](https://opendatascience.com/xgboost-enhancement-over-gradient-boosting-machines/?utm_campaign=Learning%20Posts&utm_content=111061559&utm_medium=social&utm_source=twitter&hss_channel=tw-3018841323) introduzindo uma das bibliotecas usadas para aplicar o gradient boosting chamado XGBoost.
- Se você está interessado em aprender como fazer o design e construir aplicações de aprendizado de máquina e levá-las à produção, Emmanual Ameisen cobre isso com esse [livro](https://www.amazon.com/Building-Machine-Learning-Powered-Applications/dp/149204511X/).
----------
*Se você tem uma história que gostaria de ser publicada na próxima edição da Newsletter de NLP, por favor mande um email para ellfae@gmail ou me mande uma mensagem via* [*Twitter*](https://twitter.com/omarsar0)*.*
---
## File: posts/_posts/2020-01-12-NLP_Newsletter_Tokenizers,_TensorFlow_2_1,_TextVe.md
---
layout: post
title: "NLP Newsletter #1 [EN]: Tokenizers, TensorFlow 2.1, TextVectorization, TorchIO, NLP Shortfalls,…"
author: billy_rick
modified:
comments: true
excerpt: "This edition covers a series of improved NLP pipelines, NLP shortfalls, an AI system for breast cancer screening,…"
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_1.png
---
\\
Hello and Happy New Year! Due to numerous requests, I have decided to bring back the **NLP Newsletter**. This time I will keep it short and focused (also maintained in this [repo](https://github.com/dair-ai/nlp_newsletter)). The objective of this newsletter is to keep you informed on some of the interesting and recent stories related to NLP and ML (across a few categories) without taking too much time from your busy day.
# Publications 📙
***AI system for breast cancer screening***
\\
DeepMind published a new paper in Nature titled “[International evaluation of an AI system for breast cancer screening](https://www.nature.com/articles/s41586-019-1799-6)”. According to the authors, the work is about the evaluation of an AI system that surpasses human experts in breast cancer screening. Whether this is actually achievable by current AI systems is still up for debate and there has been continuous criticism on this type of system and how they are evaluated. Here is [short summary](https://www.nature.com/articles/d41586-019-03822-8) of the paper.
\\
***Information Extraction***
\\
Pankaj Gupta publicly released his Ph.D. Thesis titled “[Neural Information Extraction From Natural Language Text](https://www.researchgate.net/publication/336739252_PhD_Thesis_Neural_Information_Extraction_From_Natural_Language_Text)”. The main discussion is how to efficiently extract the semantic relationships from natural language text using neural-based approaches. Such research effort aims to contribute to building structured knowledge bases, that can be used in a series of downstream NLP applications such as web search, question-answering, among other tasks.
\\
***Improved recommendations***
\\
Researchers at MIT and IBM developed a [method](http://news.mit.edu/2019/finding-good-read-among-billions-of-choices-1220) (published at NeurIPS last year) for categorizing, surfacing, and searching relevant documents based on a combination of three widely-used text-analysis tools: *topic modeling*, *word embeddings*, and *optimal transport*. The method also gives promising results for sorting documents. Such methods are applicable in a wide variety of scenarios and applications that require improved and faster suggestions on large-scale data such as search and recommendation systems.
# ML and NLP Creativity and Society 🎨
***AI careers***
\\
The 2019 AI Index [report](https://hai.stanford.edu/sites/g/files/sbiybj10986/f/ai_index_2019_report.pdf) suggests that there is more demand that there is a supply of AI practitioners. However, there are various aspects of AI-related jobs such as career transitions and interviews that are still not properly defined.
\\
In this [post](https://towardsdatascience.com/how-i-found-my-current-job-3fb22e511a1f), Vladimir Iglovivok goes into great detail on his career and ML adventure from building traditional recommender systems to building spectacular computer vision models that won competitions on Kaggle. He now works on autonomous vehicles at Lyft but the [journey](https://towardsdatascience.com/how-i-found-my-current-job-3fb22e511a1f) of getting there wasn’t so easy.
\\
If you are really interested and serious about a career in AI, Andrew Ng’s company, deeplearning.ai, founded Workera, which aims to specifically help data scientists and machine learning engineers with their AI careers. Obtain their official [report](https://workera.ai/candidates/report) here.
# ML/NLP Tools and Datasets ⚙️
***An ultra-fast tokenizer***
\\
Hugging Face, the NLP startup behind Transformers, has open-sourced Tokenizers, an ultra-fast implementation of tokenization that can be used in modern NLP pipelines. Check out the [GitHub repo](https://github.com/huggingface/tokenizers) for the documentation on how to use Tokenizers.
\\
*Tokenizers — Python bindings*
\\
TensorFlow 2.1 incorporates a new [TextVectorization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization) layer which allows you to easily deal with raw strings and efficiently perform text normalization, tokenization, n-grams generation, and vocabulary indexing. Read the release here and check out Chollet’s [Colab notebook](https://colab.research.google.com/drive/1RvCnR7h0_l4Ekn5vINWToI9TNJdpUZB3) demonstrating how to use the feature for end-to-end text classification.
\\
***NLP and ML for Search***
\\
One of the fields that made tremendous progress this past year was NLP with a range of improvements and new research directions. One of those domains that could potentially benefit from transfer learning NLP is *search*.
\\
Although search belongs to the field of information retrieval there is an opportunity to build search engines that improve semantic search using modern NLP techniques such as contextualized representations from a transformer-based model like [BERT](https://arxiv.org/abs/1810.04805). Google released a [blog post](https://www.blog.google/products/search/search-language-understanding-bert/) a couple of months back discussing how they are leveraging BERT models for improving and understanding searches.
\\
If you are curious about how contextualized representations can be applied to search using open-search technologies such as Elasticsearch and TensorFlow, you can take a look at either this [post](https://towardsdatascience.com/elasticsearch-meets-bert-building-search-engine-with-elasticsearch-and-bert-9e74bf5b4cf2) or this [one](https://towardsdatascience.com/building-a-search-engine-with-bert-and-tensorflow-c6fdc0186c8a).
\\
***Medical image analysis***
\\
[TorchIO](https://github.com/fepegar/torchio) is a Python package based on the popular deep learning library called PyTorch. TorchIO offers functionalities to easily and efficiently read and sample 3D medical images. Features include spatial transforms for data augmentation and preprocessing.
\\
[source](https://github.com/fepegar/torchio)
# Ethics in AI 🚨
***Fraudulent behavior in ML community***
\\
This just came in! 1st Place winners of a Kaggle contest were disqualified for fraudulent activity. The team used clever but irresponsible and unacceptable tactics to win first place in the competition. Here is the [full story](https://www.kaggle.com/c/petfinder-adoption-prediction/discussion/125436). This story highlights one in many of the serious and unacceptable behaviors that the machine learning community wants to mitigate. The proper and ethical use of ML technologies is the only way forward.
\\
***Gender bias in machine translation***
\\
On the topic of whether machine translation systems reflect gender bias, a group of researchers published this excellent [paper](https://arxiv.org/abs/1809.02208) presenting a case study using Google Translate. One of the findings claimed by the authors is that Google Translate “exhibits a strong tendency towards male defaults, in particular for fields linked to unbalanced gender distribution such as STEM jobs.”
\\
***ML Bias and Fairness***
\\
If you want to get all caught up with AI ethics and fairness, this is a nice [podcast episode](https://twimlai.com/twiml-talk-336-trends-in-fairness-and-ai-ethics-with-timnit-gebru/) featuring Timnit Gebru and hosted by TWIML.
\\
Timnit is a prominent researcher in ML fairness who together with Eun Seo Jo, published a [paper](https://arxiv.org/abs/1912.10389) where they identify five key approaches in document collection practices in archives that can provide more reliable methods for data collection in sociocultural ML. This could potentially lead to more systematic data collection methods gained from interdisciplinary collaborative research.
\\
Sina Fazelpour and Zachary Lipton recently published a [paper](http://zacklipton.com/media/papers/fairness-non-ideal-fazelpour-lipton-2020.pdf) where they argue that due to the nature of how our non-ideal world arose it is possible that fair ML based on the ideal thinking can potentially lead to misguided policies and interventions. In fact, their analysis demonstrates “that shortcomings of proposed fair ML algorithms reflect broader troubles faced by the ideal approach.”
# Articles and Blog posts ✍️
***NLP shortfalls***
\\
Benjamin Heinzerling published an interesting article in The Gradient where he discusses areas where NLP falls short such as argument comprehension and commonsense reasoning. Benjamin makes reference to a recent [paper](https://www.aclweb.org/anthology/P19-1459/) by Nivin & Kao that challenges and questions the capabilities of transfer learning and language models for high-level natural language understanding. [Read](https://thegradient.pub/nlps-clever-hans-moment-has-arrived/) more about this excellent summary of the analysis performed in the research.
\\
***2019 NLP and ML Highlights***
\\
For the new year, I released a [report](https://medium.com/dair-ai/nlp-year-in-review-2019-fb8d523bcb19) documenting some of the most interesting NLP and ML highlights that I came across in 2019.
Sebastian Ruder also recently wrote an excellent and detailed [blog post](https://ruder.io/research-highlights-2019/) about the top ten ML and NLP research directions that he found impactful in 2019. Among the list are topics such as universal unsupervised pretraining, ML and NLP applied to science, augmenting pretrained models, efficient and long-range Transformers, among others.
\\
*“VideoBERT (*[*Sun et al., 2019*](https://arxiv.org/abs/1904.01766)*), a recent multimodal variant of BERT that generates video “tokens” given a recipe (above) and predicts future tokens at different time scales given a video token (below).” —* [*source*](https://arxiv.org/pdf/1904.01766.pdf)
\\
Google AI Research publishes a [summary](https://ai.googleblog.com/2020/01/google-research-looking-back-at-2019.html) of the research they conducted over the year and the future research directions they are paying attention to.
# ML/NLP Education 🎓
***Democratizing AI education***
\\
In an effort to democratize AI education and to educate the masses about the implications of AI technology, the University of Helsinki partnered with Reaktor to release a brilliant free course covering AI fundamentals. The [popular course](https://www.elementsofai.com/) is called “Elements of AI” and includes topics such as AI ethics, AI philosophy, neural networks, Naive Bayes rule, among other foundational topics.
\\
Stanford CS224N is back with another [iteration](http://web.stanford.edu/class/cs224n/) of the popular “Natural Language Processing with Deep Learning” course. The course officially started January 7 of this year so if you want to follow, go to their website for the full syllabus, slides, videos, paper reading suggestions, etc.
\\
***Top NLP and ML Books***
\\
I tweeted my top book recommendations for theoretical and practical NLP and ML, it was well-received. I would like to share that list here via the tweet: https://twitter.com/omarsar0/status/1214547402838986754?s=20
\\
***Machine Learning with Kernel Methods***
\\
Kernel methods such as PCA and K-means have been around for quite some time and that’s because they have been successfully applied for a wide variety of applications such as graphs and biological sequences. Check out this comprehensive set of [slides](http://members.cbio.mines-paristech.fr/~jvert/svn/kernelcourse/slides/master2017/master2017.pdf) covering a wide range of kernel methods and their inner workings. Here is also a great [blog](https://francisbach.com/cursed-kernels/) (maintained by Francis Bach) discussing aspects of kernel methods and other machine learning topics.
# Notable Mentions ⭐️
Here is a list of noteworthy stories that are worth your attention:
- John Langford runs this incredible [blog](https://hunch.net/) discussing machine learning theory
- Many of the industry ML-oriented technologies have been using Gradient Boosting machines for years. Check out this [post](https://opendatascience.com/xgboost-enhancement-over-gradient-boosting-machines/?utm_campaign=Learning%20Posts&utm_content=111061559&utm_medium=social&utm_source=twitter&hss_channel=tw-3018841323) introducing one of the libraries used to apply gradient boosting called XGBoost.
- If you are interested in learning how to design and build machine learning-powered applications and take them to production, Emmanuel Ameisen has you covered with this [book](https://www.amazon.com/Building-Machine-Learning-Powered-Applications/dp/149204511X/).
----------
*If you have a story that you would like to see hosted in the next edition of the NLP Newsletter, please send me an email to ellfae@gmail.com or send me a message via* [*Twitter*](https://twitter.com/omarsar0)*.*
---
## File: posts/_posts/2020-01-19-NLP_Newsletter[PT-BR]_Reformer,_DeepMath,_ELECTRA,_TinyB.md
---
layout: post
title: "NLP Newsletter #2 [PT-BR]: Reformer, DeepMath, ELECTRA, TinyBERT para busca, VizSeq, Open-Sourcing ML,…"
author: fclesio
modified:
comments: true
excerpt: "Esta segunda newsletter aborda topics que vão de interpretabilidade de modelos para enovelamento de proteínas (protein folding) até active transfer learning"
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_2.png
---
\\
Bem vindo novamente à NLP Newsletter! 👋 Esta segunda newsletter aborda topicos que vão de interpretabilidade de modelos para enovelamento de proteínas (protein folding) até active transfer learning. *Você pode encontrar a versão Markdown desta edição no final.*
# Publicações 📙
***Confiando na incerteza dos modelos***
\\
Um artigo recente do Google AI, publicado na NeurIPS, analisa se as probabilidades de um modelo refletem a sua capacidade de prever dados fora de distribuição ou mudança no conjunto de dados (shifted data).
\\
Ensembles profundos (Deep ensembles) tiveram um melhor desempenho (_i.e._, melhoraram a incerteza do modelo) no dataset com mudança no conjunto de dados (data shift), enquanto outros modelos não tornaram-se mais incertos com o mesmo data shift, mas ao contrario transformaram-se confidentemente errados(Leia o paper [aqui](https://arxiv.org/abs/1906.02530) e o sumário [aqui](https://ai.googleblog.com/2020/01/can-you-trust-your-models-uncertainty.html).).
\\
*Corrupção da imagem —* [*fonte*](https://ai.googleblog.com/2020/01/can-you-trust-your-models-uncertainty.html)
\\
***Generalização Sistemática***
\\
Um [trabalho interessante](https://www.semanticscholar.org/paper/Systematic-Generalization%3A-What-Is-Required-and-Can-Bahdanau-Murty/6c7494a47cc5421a7b636c244e13586dc2dab007) publicado na ICLR apresenta uma comparação entre modelos modulares e modelos genéricos e as suas respectivas efetividades para *generalização sistematica* em entendimento de linguagem (language understanding). Com base na avaliação do _reasoning_ realizada em uma tarefa em que se [respondia uma pergunta visual](https://arxiv.org/abs/1909.01860), os autores concluem que pode haver a necessidade de regularizadores e priorizadores explícitos para se conseguir uma generalização sistemática.
\\
***Um modelo eficiente baseado em Transformer chamado Reformer***
\\
É bem conhecido que um modelo Transformer é bastante limitado em relação à janela de contexto que pode ser coberta, devido aos cálculos de algo custo computacional realizados na camada de [Attention](https://mlexplained.com/2017/12/29/attention-is-all-you-need-explained/). Assim, só pode ser possível aplicar o modelo Transformer a tamanhos de texto limitados ou gerar frases curtas e/ou peças musicais. O GoogleAI publicou recentemente uma variante eficiente de um modelo Transformer chamado [Reformer](https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html). O foco principal deste método é ser capaz de lidar com janelas de contexto muito mais altas e, ao mesmo tempo, reduzir os requisitos computacionais com a melhoria da eficiência do uso de memória. O Reformer usa o "locality-sensitive-hashing" ([LSH](https://en.wikipedia.org/wiki/Locality-sensitive_hashing)) para agrupar vetores similares e criar segmentos a partir deles, o que permite o processamento paralelo. A camada de Attention é então aplicada a estes segmentos menores e em partes vizinhas correspondentes - isto é, o que reduz a carga computacional. A eficiência de memória é obtida usando camadas reversíveis que permitem que as informações de entrada de cada camada sejam recalculadas sob demanda durante o treinamento via backpropagation. Esta é uma técnica simples que evita que o modelo tenha a necessidade de armazenar as ativações em memória. Confira este [notebook no Colab](https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/image_generation.ipynb) para ver como um modelo Reformer pode ser aplicado a uma tarefa de geração de imagens.
\\
*"Locality-sensitive-hashing: O Reformer assume uma sequência de palavras de entrada, onde cada palavra é na verdade um vetor representando palavras individuais (ou pixels, no caso das imagens) na primeira camada e contextos maiores nas camadas subsequentes. O LSH é aplicado à sequência, depois que as palavras são ordenadas pelo seu hash e particionadas. O Attention é aplicado apenas dentro de um único pedaço e dos seus vizinhos imediatos". - [fonte](https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html)*
\\
***Adaptação de Dominio de forma não-supervisionada para Classificação de Textos***
\\
Este [trabalho](https://arxiv.org/abs/2001.04362) propõe uma combinação de medidas de distância incorporadas em uma função de perda adicional, para treinar um modelo e melhorar a adaptação não-supervisionada do domínio. O modelo é estendido a um DistanceNet Bandit que otimiza os resultados para "transferência para o domínio alvo de poucos recursos". O principal problema abordado com este método é como lidar com a disparidade entre os dados de diferentes domínios, especificamente no que diz respeito à tarefas de NLP, como a análise de sentimentos (sentiment analysis).
\\
***Melhoria de Representações Contextualizadas***
\\
Este [artigo](https://openreview.net/forum?id=r1xMH1BtvB) propõe uma tarefa de pré-treino mais eficiente em termos de amostragem chamada *detecção de token*. Esta tarefa pode ser utilizada para treinar um modelo linguístico que é mais eficiente do que métodos de pré-treino com modelagem de linguagem mascarada, como o BERT. O modelo é chamado ELECTRA e suas representações contextualizadas superam as do BERT e XLNET com os mesmos dados e tamanho de modelo. O método funciona particularmente bem no regime de baixa computação. Este é um esforço para construir modelos de linguagem menores e mais baratos.
\\
***Interpretabilidade de Modelos***
\\
A publicação mais recente da Distill intitulada "[Visualizing the Impact of Feature Attribution Baselines](https://distill.pub/2020/attribution-baselines/)" discute os [gradientes integrados](https://medium.com/@kartikeyabhardwaj98/integrated-gradients-for-deep-neural-networks-c114e3968eae) que são usados para interpretar redes neurais em vários problemas, identificando quais recursos são relevantes para prever um determinado ponto. O problema é definir e preservar corretamente uma noção de *falta* que é o que se pretende com a entrada de base dos gradientes integrados. O desafio aqui, no contexto da interpretabilidade de modelos, é que o método deve assegurar que o modelo não destaque as características em falta como importantes, evitando ao mesmo tempo dar zero importância às entradas de _baseline_, o que pode facilmente acontecer. O autor propõe avaliar quantitativamente os diferentes efeitos de algumas escolhas de _baseline_ previamente utilizadas e propostas que melhor preservem a noção de falta.
# Criatividade e Sociedade 🎨
***Incompatibilidade de sentimentos***
\\
Este [estudo longitudinal](https://ieeexplore.ieee.org/abstract/document/8952437) descobre que a emoção extraída através do uso de algoritmos baseados em texto, muitas vezes não é a mesma que as emoções auto-relatadas.
\\
***Compreensão da dopamina e o enovelamento de proteínas (protein folding)***
\\
A DeepMind lançou recentemente **dois** artigos interessantes na revista Nature. O primeiro [artigo](https://deepmind.com/blog/article/Dopamine-and-temporal-difference-learning-A-fruitful-relationship-between-neuroscience-and-AI) visa entender melhor como funciona a dopamina no cérebro usando a aprendizagem por reforço. O segundo [paper](https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery) está mais relacionado com [enovelamento de proteínas](https://en.wikipedia.org/wiki/Protein_folding) e tenta compreendê-lo melhor para ser capaz de potencialmente descobrir tratamentos para uma ampla gama de doenças. Estes são grandes exemplos de como sistemas de IA poderiam potencialmente ser aplicados em aplicações do mundo real para ajudar a sociedade.
\\
*“Formas 3D complexas emergem de um conjunto de aminoácidos.” —* [*fonte*](https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery)
\\
***Entrevistas sobre o ML na sociedade***
\\
Em uma [entrevista](https://www.youtube.com/watch?v=I-EIVlHvHRM&feature=youtu.be) com a Wired, Refik Anadol discute o potencial dos algoritmos de aprendizagem de máquina para criação de arte. Este é um excelente exemplo de como Machine Learning pode ser usado de forma a criativa.
\\
Um dos setores em que a IA pode ter um grande impacto é na educação. Em um novo [episódio](https://engineering.stanford.edu/magazine/article/emma-brunskill-amped-education-ai?sf115875862=1), que faz parte de "_The Future of Everything_", Russ Altman e Emma Brunskill têm uma discussão profunda sobre a aprendizagem assistida por computador.
# Ferramentas e Datasets ⚙️
***Modelos PyTorch em produção***
\\
O Cortex é uma ferramenta para automatizar a infra-estrutura e implementar modelos PyTorch como APIs em produção com AWS. Saiba mais sobre como isso é feito [aqui](https://medium.com/pytorch/how-to-build-production-software-with-pytorch-9a8725382f2a).
\\
***Visualizando Sequências de Geração de Texto***
\\
O time do Facebook AI lançou o [VizSeq](https://ai.facebook.com/blog/vizseq-a-visual-analysis-toolkit-for-accelerating-text-generation-research/), que é uma ferramenta que auxilia na avaliação visual de seqüências de geração de texto sob métricas como BLUE e METEOR. O principal objetivo desta ferramenta é fornecer uma análise mais intuitiva dos conjuntos de dados de texto, alavancando visualizações e tornando-as mais escaláveis e produtivas para todos os pesquisadores. Leia o artigo completo [aqui](https://www.aclweb.org/anthology/D19-3043.pdf).
\\
[*Fonte*](https://ai.facebook.com/blog/vizseq-a-visual-analysis-toolkit-for-accelerating-text-generation-research/)
\\
***Estado da arte em reconhecimento de fala online***
\\
O FacebookAI tornou open-source o [wav2letter@anywhere](https://ai.facebook.com/blog/online-speech-recognition-with-wav2letteranywhere/) que é um framework de inferência que é baseado em um modelo acústico que usa o Transformer como base para reconhecimento de fala online de última geração. Grandes melhorias estão em torno do tamanho do modelo e reduzindo a latência entre o áudio e a transcrição, o que é importante para seja obtida uma inferência mais rápida em tempo real.
\\
*Processamento de fala —* [*fonte*](https://ai.facebook.com/blog/online-speech-recognition-with-wav2letteranywhere/)
# Ética em IA 🚨
***Implicações de IA***
\\
Em um esforço para evitar abusos e ações antiéticas dos sistemas de IA sobre o público, a União Européia está considerando proibir a tecnologia de reconhecimento facial do público por cinco anos. ([História completa](https://www.reuters.com/article/us-eu-ai/eu-mulls-five-year-ban-on-facial-recognition-tech-in-public-areas-idUSKBN1ZF2QL))
\\
***Os custos ambientais de NLP***
\\
Talvez negligenciado na maioria das vezes, este [documento](https://arxiv.org/abs/1906.02243) discute as considerações energéticas e políticas para abordagens modernas de deep learning em NLP. É amplamente conhecido que os modelos atuais dependem de bilhões de parâmetros e que, por sua vez, dependem de grandes recursos computacionais que demandam um consumo substancial de energia. Os autores esperam espalhar mais consciência sobre os custos ambientais envolvidos no treinamento desses modernos modelos de NLP.
\\
Zachary Lipton discute equidade(fairness), interpretabilidade e os perigos do solucionismo em Machine Learning nesta [palestra](https://c4ejournal.net/2020/01/16/zack-lipton-fairness-interpretability-and-the-dangers-of-solutionism-ethics-of-ai-in-context2020-c4ej-2/) proferida na Universidade de Toronto. Os principais tópicos giraram em torno de considerações cuidadosas e implicações das abordagens de equidade em ML.
# Artigos e posts de blogs ✍️
***Open-Sourcing ML***
\\
Thomas Wolf, líder científico da Hugging Face, compartilha excelentes conselhos para aqueles que planejam fazer código ML open-source ou pesquisa. Encontre o tópico do Twitter [aqui](https://twitter.com/Thom_Wolf/status/1216990543533821952?s=20).
\\
***Introdução para aprendizagem auto-supervisionada para computer vision****
\\
Jeremy Howard escreveu este grande [blog post](https://www.fast.ai/2020/01/13/self_supervised/) introduzindo brevemente o conceito de aprendizagem auto-supervisionada no contexto de computer vision. Eu adoro estes pequenos resumos, pois ajudam a dar uma introdução confiável no caso de você estar interessado em aplicar técnicas deste domínio ao seu próprio problema.
\\
***TinyBERT para problemas de busca***
\\
Já vimos o sucesso de muitas variantes de modelos BERT (por exemplo, [DistilBERT](https://medium.com/huggingface/distilbert-8cf3380435b5)) que utilizam alguma forma de [destilação do conhecimento](https://nervanasystems.github.io/distiller/knowledge_distillation.html) para diminuir substancialmente o tamanho do modelo e melhorar a velocidade. Algumas pessoas usaram uma variante do BERT chamada, [TinyBERT](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT), e aplicaram-na a uma [solução de busca baseada em palavras-chave](https://towardsdatascience.com/tinybert-for-search-10x-faster-and-20x-smaller-than-bert-74cd1b6b5aec). Este projeto foi inspirado por esta [solução de busca](https://www.blog.google/products/search/search-language-understanding-bert/) para compreender as buscas propostas pelo Google. A maior parte da arquitetura que ela funciona é em uma CPU padrão e pode ser usada para melhorar e entender os resultados das buscas.
\\
***Active Transfer Learning***
\\
Rober Monarch escreveu este excelente [blog post](https://medium.com/pytorch/https-medium-com-robert-munro-active-learning-with-pytorch-2f3ee8ebec) sobre _Active Transfer Learning_ que faz parte de seu próximo livro chamado [Human-in-the-loop Machine Learning](https://www.manning.com/books/human-in-the-loop-machine-learning). Ele está escrevendo ótimos posts em seu blog sobre métodos para combinar inteligência humana e máquinas para resolução de problemas. Ele também fornece implementações em PyTorch dos métodos discutidos.
\\
***Revelando os segredos ocultos do BERT***
\\
Anna Roger escreveu este divertido e interessante [blog post](https://text-machine-lab.github.io/blog/2020/bert-secrets/) que fala sobre o que realmente acontece com um BERT otimizado, e se os alegados pontos fortes são usados para abordar tarefas posteriores, tais como análise de sentimentos, vinculação textual e inferência da linguagem natural, entre outras. Os resultados das análises propostas sugerem que o BERT está excessivamente superparametrizado (overparameterized) e que os benefícios identificados do componente de _self-attention_ da estrutura podem não ser necessariamente benéficos como se imagina. Em particular no que diz respeito à informação linguística que está sendo codificada e utilizada para a parte de inferência.
# Educação 🎓
***Redes Neurais para NLP***
\\
Graham Neubig, professor de PNL na CMU, tem [lançado alguns vídeos](https://www.youtube.com/playlist?list=PL8PYTP1V4I8CJ7nMxMC8aXv8WqKYwj-aJ) para a aula "Redes Neurais para NLP" que está sendo ministrada neste semestre. Eu recomendo altamente esta playlist para aqueles interessados em aprender sobre os métodos modernos de NLP.
\\
***Deep Learning Math (DeepMath)***
\\
Quer mergulhar profundamente na matemática por trás dos métodos de deep learning? Aqui está uma [série de vídeo-palestras](https://www.youtube.com/playlist?list=PLWQvhvMdDChzsThHFe4lYAff3pu2m0v2H) com uma vasta gama de palestrantes.
\\
***Cursos de Python e Tutoriais***
\\
Python tornou-se uma das linguagens de programação mais requisitadas não só na indústria de TI, mas também no espaço da ciência dos dados. Em um esforço para qualificar alunos de todo o mundo com conhecimentos práticos de Python, o Google lançou o "Google IT Automation with Python Professional Certificate". Saiba mais sobre o lançamento [aqui](https://blog.google/outreach-initiatives/grow-with-google/new-certificate-help-people-grow-careers) e veja o curso [aqui](https://www.coursera.org/professional-certificates/google-it-automation). Embora o curso não esteja diretamente relacionado ao ML ou à IA, é definitivamente um bom curso de fundamentos para deseja tornar-se proficiente com a linguagem Python. Bolsas de estudo também estão disponíveis.
\\
Aqui está outra [série de vídeos](https://www.youtube.com/watch?v=fMqL5vckiU0&list=PL-wATfeyAMNrtbkCNsLcpoAyBBRJZVlnf) promissora chamada "Deep Learning (for Audio) with Python" com foco no uso de Tensorflow e Python para construção de aplicações relacionadas a áudio/música, e alavancando o uso de deep learning.
\\
[*fonte*](https://www.youtube.com/watch?v=fMqL5vckiU0&list=PL-wATfeyAMNrtbkCNsLcpoAyBBRJZVlnf)
\\
Andrew Trask lançou um conjunto de [notebooks com tutoriais passo-a-passo](https://c4ejournal.net/2020/01/16/zack-lipton-fairness-interpretability-and-the-dangers-of-solutionism-ethics-of-ai-in-context2020-c4ej-2/) para alcançar deep learning de forma descentralizada com objetivo de preservação da privacidade. Todos os notebooks contêm implementações PyTorch e são destinados a iniciantes.
\\
***Estado da arte em Deep Learning***
\\
Confira esta [palestra em vídeo](https://www.youtube.com/watch?v=0VH1Lim8gL8) de Lex Fridman sobre a recente pesquisa e desenvolvimento em Deep Learning. Ele fala sobre os principais avanços em tópicos como perceptrons, redes neurais, backpropagation, CNN, deep learning, ImageNet, GANs, AlphaGo, e Transformers. Esta palestra faz parte da Série de Deep Learning do MIT.
\\
***Online learning e pesquisa***
\\
Há muitas e grandes iniciativas online para colaboração tanto em pesquisa quanto em aprendizagem. Os meus favoritos são a sessão de leitura de matemática [MLT's](https://twitter.com/__MLT__) e este novo esforço de colaboração distribuída em pesquisa AI iniciado por [nightai](https://www.nightai.co/). Recentemente, tem havido muitos grupos de estudo como este online e estes grupos de estudos são ótimas maneiras de mergulhar no mundo do ML.
\\
***Perspectivas em aprendizagem por reforço***
\\
Aprenda com a Dra. Katja Hofmann os principais conceitos e métodos de aprendizagem por reforço nesta [série de webinars](https://note.microsoft.com/MSR-Webinar-RL-Algorithm-to-Adoption-Registration-Live.html?wt.mc_id=twitter_MSR-WBNR_post_v3).
# Menções honrosas ⭐️
Confira [esta implementação limpa e auto-contida em PyTorch](https://gist.github.com/y0ast/d91d09565462125a1eb75acc65da1469) da ResNet-18 aplicada ao CIFAR-10, que atinge ~94% de precisão.
\\
PyTorch 1.4 é lançado! Confira as notas de lançamento [aqui](https://github.com/pytorch/pytorch/releases/tag/v1.4.0).
\\
Elona Shatri escreveu este excelente [resumo](_COPY11@e.shatri1/what-is-optical-music-recognition-6515d8a53e01) sobre como ela pretende abordar o reconhecimento da música óptica usando deep learning.
\\
O título para este post no blog é auto-explicativo: ["The Case for Bayesian Deep Learning"](https://cims.nyu.edu/~andrewgw/caseforbdl/)".
\\
Chris Said compartilha sua [experiência](https://chris-said.io/2020/01/10/optimizing-sample-sizes-in-ab-testing-part-I/) na otimização de tamanhos de amostras para testes A/B, uma parte importante em se tratando de Data Science de forma aplicada. Os tópicos incluem os custos e benefícios de grandes tamanhos de amostras e melhores práticas para os profissionais.
\\
Neural Data Server (NDS) is a dedicated search engine for obtaining transfer learning data. Read about the method [here](https://arxiv.org/abs/2001.02799) and the service [here](http://aidemos.cs.toronto.edu/nds/).
O Neural Data Server (NDS) é um mecanismo de busca dedicado à obtenção de dados via _transfer learning_. Leia sobre o método [aqui](https://arxiv.org/abs/2001.02799) e o serviço [aqui](http://aidemos.cs.toronto.edu/nds/).
---
## File: posts/_posts/2020-01-19-NLP_Newsletter_Reformer,_DeepMath,_ELECTRA,_TinyB copy.md
---
layout: post
title: "NLP Newsletter #2 [EN]: Reformer, DeepMath, ELECTRA, TinyBERT for Search, VizSeq, Open-Sourcing ML,…"
author: billy_rick
modified:
comments: true
excerpt: "This second issue covers topics that range from model interpretability to protein folding to active transfer learning."
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_2.png
---
\\
Welcome back to the NLP Newsletter! 👋 This second issue covers topics that range from model interpretability to protein folding to active transfer learning. *You can find the Markdown version of this edition towards the end.*
# Publications 📙
***On trusting model’s uncertainty***
\\
A recent paper from Google AI, published at NeurIPS, looks at whether a model’s probabilities reflect its ability to predict out-of-distribution and shifted data. Deep ensembles were found to perform better (i.e., improved model uncertainty) on the dataset shift while other models did not become increasingly unsure with dataset shift, but instead became confidently wrong. (Read the paper [here](https://arxiv.org/abs/1906.02530) and the summary [here](https://ai.googleblog.com/2020/01/can-you-trust-your-models-uncertainty.html).)
\\
*image corruption —* [*source*](https://ai.googleblog.com/2020/01/can-you-trust-your-models-uncertainty.html)
\\
***Systematic generalization***
\\
An interesting [work](https://www.semanticscholar.org/paper/Systematic-Generalization%3A-What-Is-Required-and-Can-Bahdanau-Murty/6c7494a47cc5421a7b636c244e13586dc2dab007) published in ICLR presents a comparison between modular models and generic models and their effectiveness for *systematic generalization* in language understanding. Based on a reasoning evaluation performed on a [visual question answering](https://arxiv.org/abs/1909.01860) task, the authors conclude that there may be a need for explicit regularizers and priors to achieve systematic generalization.
\\
***An efficient transformer-based model called Reformer***
\\
It is well known that a Transformer model is quite limited in the context window it can cover due to the expensive computations performed in the attention layer. Thus, it may only be possible to apply the Transformer model to limited text sizes or generate short statements or pieces of music. GoogleAI recently published an efficient variant of a Transformer model called [Reformer](https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html). The main focus of this method is to be able to deal with much higher context windows while at the same time reducing computational requirements and improving memory efficiency. Reformer uses locality-sensitive-hashing ([LSH](https://en.wikipedia.org/wiki/Locality-sensitive_hashing)) to group similar vectors together and creates segments out of them, which enable parallel processing. The attention is then applied to these smaller segments and corresponding neighboring parts — this is what reduces the computational load. Memory efficiency is accomplished using reversible layers that allow the input information of each layer to be recomputed on-demand while training via backpropagation. This is a simple technique that avoids the model of the need to store activations in memory. Check out this Colab [notebook](https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/image_generation.ipynb) to see how a Reformer model can be applied to an image generation task.
\\
*“Locality-sensitive-hashing: Reformer takes in an input sequence of keys, where each key is a vector representing individual words (or pixels, in the case of images) in the first layer and larger contexts in subsequent layers. LSH is applied to the sequence, after which the keys are sorted by their hash and chunked. Attention is applied only within a single chunk and its immediate neighbors.” — [source](https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html)*
\\
***Unsupervised Domain Adaptation for Text Classification***
\\
This [work](https://arxiv.org/abs/2001.04362) proposes a combination of distance measures incorporated into an additional loss function to train a model and improve unsupervised domain adaptation. The model is extended to a DistanceNet Bandit model that optimizes results “for transfer to the low-resource target domain”. The key problem being addressed with this method is how to deal with the dissimilarity between data from different domains, specifically as it relates to NLP tasks such as sentiment analysis.
\\
***Improved Contextualized Representations***
\\
This [paper](https://openreview.net/forum?id=r1xMH1BtvB) proposes a more sample-efficient pretraining task called *token detection* for training a language model that is more efficient than masked language modeling pretraining methods such as BERT. The model is coined ELECTRA and its contextualized representations outperform those of BERT and XLNET on the same data and model size. The method particularly works well on the low-compute regime. This is an effort to build smaller and cheaper language models.
\\
***Model Interpretability***
\\
Distill recent publication titled “[Visualizing the Impact of Feature Attribution Baselines](https://distill.pub/2020/attribution-baselines/)” discusses [integrated gradients](https://medium.com/@kartikeyabhardwaj98/integrated-gradients-for-deep-neural-networks-c114e3968eae) that are used to interpret neural networks in various problems by identifying which features are relevant to predict a certain data point. The problem is to properly define and preserve a notion of *missingness* which is what the baseline input of integrated gradients is intended for. The challenge here, in the context of model interpretability, is that the method must ensure that the model doesn’t highlight missing features as important while at the same time avoiding giving the baseline inputs zero importance which can easily happen. The author proposes to quantitatively evaluate the different effects of some previously used and proposed baseline choices that better preserve the notion of missingness.
# Creativity and Society 🎨
***Sentiment mismatch***
\\
This longitudinal [study](https://ieeexplore.ieee.org/abstract/document/8952437) finds that emotion extracted via the use of text-based algorithms are often not the same as the self-reported emotions.
\\
***Dopamine understanding and protein folding***
\\
DeepMind recently released **two** interesting papers in Nature. The first [paper](https://deepmind.com/blog/article/Dopamine-and-temporal-difference-learning-A-fruitful-relationship-between-neuroscience-and-AI) aims to better understand how dopamine in the brain works using reinforcement learning. The second [paper](https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery) is more related to [protein folding](https://en.wikipedia.org/wiki/Protein_folding) and trying to understand it better to be able to potentially discover treatments for a wide range of diseases. These are great examples of how AI systems could potentially be applied to real-world applications to help society.
\\
*“Complex 3D shapes emerge from a string of amino acids.” —* [*source*](https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery)
\\
***Interviews about ML in society***
\\
In an [interview](https://www.youtube.com/watch?v=I-EIVlHvHRM&feature=youtu.be) with Wired, Refik Anadol discusses the potential of machine learning algorithms to create beautiful art. This is an excellent example of how ML can be used for creativity.
\\
One of the sectors where AI could have a major impact is in education. In a new [episode](https://engineering.stanford.edu/magazine/article/emma-brunskill-amped-education-ai?sf115875862=1), which is part of “The Future of Everything”, Russ Altman and Emma Brunskill have a deep discussion about computer-assisted learning.
# Tools and Datasets ⚙️
***PyTorch models in production***
\\
Cortex is a tool to automate the infrastructure and deploy PyTorch models as APIs in production with AWS. Learn more about how it’s done [here](https://medium.com/pytorch/how-to-build-production-software-with-pytorch-9a8725382f2a).
\\
***Visualizing Text Generation Sequences***
\\
Facebook AI released [VizSeq](https://ai.facebook.com/blog/vizseq-a-visual-analysis-toolkit-for-accelerating-text-generation-research/), a tool that aids in visually assessing text generation sequences under metrics like BLUE and METEOR. The main goal of this tool is to provide a more intuitive analysis of text datasets by leveraging visualizations and making it more scalable and productive for the researcher. Read the full paper [here](https://www.aclweb.org/anthology/D19-3043.pdf).
\\
[*Source*](https://ai.facebook.com/blog/vizseq-a-visual-analysis-toolkit-for-accelerating-text-generation-research/)
\\
***State-of-the-art online speech recognition***
\\
FacebookAI open-sources [wav2letter@anywhere](https://ai.facebook.com/blog/online-speech-recognition-with-wav2letteranywhere/) which is an inference framework that is based on a Transformer-based acoustic model for state-of-the-art online speech recognition. Major improvements are around the model size and reducing latency between audio and transcription, which are both important to achieving faster real-time inferencing.
\\
*speech processing —* [*source*](https://ai.facebook.com/blog/online-speech-recognition-with-wav2letteranywhere/)
# Ethics in AI 🚨
***AI implications***
\\
In an effort to prevent abuses and unethical actions of AI systems on the public, the European Union is considering to ban facial recognition technology from the public for five years. ([Full story](https://www.reuters.com/article/us-eu-ai/eu-mulls-five-year-ban-on-facial-recognition-tech-in-public-areas-idUSKBN1ZF2QL))
\\
***Environmental costs of modern NLP***
\\
Perhaps overlooked most of the time, this [paper](https://arxiv.org/abs/1906.02243) discusses the energy and policy considerations for modern deep learning approaches in NLP. It’s widely known that current models rely on billions of parameters and in turn large computational resources arising in a substantial consumption of energy. The authors hope to spread more awareness about the environmental costs involved in training these modern NLP models.
\\
Zachary Lipton discusses fairness, interpretability, and the dangers of solutionism in this [lecture](https://c4ejournal.net/2020/01/16/zack-lipton-fairness-interpretability-and-the-dangers-of-solutionism-ethics-of-ai-in-context2020-c4ej-2/) delivered at the University of Toronto. The main topics revolve around careful considerations and implications of ML fairness approaches.
# Articles and Blog posts ✍️
***Open-Sourcing ML***
\\
Thomas Wolf, science lead at Hugging Face, shares excellent advice for those planning to open-source ML code or research. Find the Twitter thread [here](https://twitter.com/Thom_Wolf/status/1216990543533821952?s=20).
\\
***Intro to self-supervised learning for computer vision***
\\
Jeremy Howard wrote this great [blog post](https://www.fast.ai/2020/01/13/self_supervised/) briefly introducing the concept of self-supervised learning in the context of computer vision. I love these short summaries as they help to give you a reliable introduction in case you are interested in applying techniques from this domain to your own problem.
\\
***TinyBERT for search***
\\
We have already seen the success of many variants of BERT models (e.g., [DistilBERT](https://medium.com/huggingface/distilbert-8cf3380435b5)) that use some form of [knowledge distillation](https://nervanasystems.github.io/distiller/knowledge_distillation.html) to substantially decrease the model size and improve speed. A few folks used a variant of BERT called, [TinyBERT](https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT), and applied it to a [keyword-based search solution](https://towardsdatascience.com/tinybert-for-search-10x-faster-and-20x-smaller-than-bert-74cd1b6b5aec). This project was inspired by this [search solution](https://www.blog.google/products/search/search-language-understanding-bert/) for understanding searches proposed by Google. The great part of the architecture is that it works on a standard CPU and can be used to improve and understand search results.
\\
***Active Transfer Learning***
\\
Rober Monarch wrote this excellent [blog post](https://medium.com/pytorch/https-medium-com-robert-munro-active-learning-with-pytorch-2f3ee8ebec) about active transfer learning which is part of his upcoming book, [Human-in-the-loop Machine Learning](https://www.manning.com/books/human-in-the-loop-machine-learning). He is currently writing great blog posts on methods for combining human and machine intelligence to solve problems. He also provides accompanying PyTorch implementations of the methods discussed.
\\
***Revealing the Dark Secrets of BERT***
\\
Anna Roger wrote this fun and interesting [blog post](https://text-machine-lab.github.io/blog/2020/bert-secrets/) which talks about what really happens with a fine-tuned BERT and whether the claimed strengths are used to approach downstream tasks such as sentiment analysis, textual entailment, and natural language inference, among others. The findings of the proposed analyses suggest that BERT is severely overparameterized and that the identified benefits of the self-attention component of the framework may not necessarily be as claimed in particular as it relates to the linguistic information that’s being encoded and used for inferencing.
# Education 🎓
***Neural Nets for NLP***
\\
Graham Neubig, NLP professor at CMU, has been [releasing videos](https://www.youtube.com/playlist?list=PL8PYTP1V4I8CJ7nMxMC8aXv8WqKYwj-aJ) for the “Neural Nets for NLP” class being delivered this semester. I highly recommend this playlist for those interested in learning about modern NLP methods.
\\
***Deep Learning Math (DeepMath)***
\\
Want to dive deep into the mathematics behind deep learning methods? Here is a [video lecture series](https://www.youtube.com/playlist?list=PLWQvhvMdDChzsThHFe4lYAff3pu2m0v2H) hosting a wide range of speakers.
\\
***Python Courses and Tutorials***
\\
Python has become one of the most in-demand programming languages not only in the IT industry but also in the data science space. In an effort to equip learners from all over the world with practical knowledge of Python, Google released the “Google IT Automation with Python Professional Certificate.” Find out more about the release [here](https://blog.google/outreach-initiatives/grow-with-google/new-certificate-help-people-grow-careers) and the course [here](https://www.coursera.org/professional-certificates/google-it-automation). Although the course is not directly related to ML or AI, it’s definitely a nice foundational course to get proficient with the Python language. Scholarships are also available.
\\
Here is another promising [video series](https://www.youtube.com/watch?v=fMqL5vckiU0&list=PL-wATfeyAMNrtbkCNsLcpoAyBBRJZVlnf) called “Deep Learning (for Audio) with Python” with a focus on using Tensorflow and Python for building applications related to audio/music by leveraging deep learning.
\\
[*source*](https://www.youtube.com/watch?v=fMqL5vckiU0&list=PL-wATfeyAMNrtbkCNsLcpoAyBBRJZVlnf)
\\
Andrew Trask released a set of [step-by-step notebook tutorials](https://github.com/OpenMined/PySyft/tree/master/examples/tutorials) for achieving privacy-preserving and decentralized deep learning. All notebooks contain PyTorch implementations and are meant for beginners.
\\
***Deep Learning State of the Art***
\\
Check out this [video lecture](https://www.youtube.com/watch?v=0VH1Lim8gL8) by Lex Fridman on the recent research and development in deep learning. He talks about major breakthroughs on topics such as perceptrons, neural networks, backpropagation, CNN, deep learning, ImageNet, GANs, AlphaGo, and the more recent Transformers. This lecture is part of the MIT Deep Learning Series.
\\
***Online learning and researching***
\\
There are many great online initiatives to collaborate in research and learning. My personal favorites are the [MLT’s](https://twitter.com/__MLT__) math reading session and this new distributed AI research collaboration effort started by [nightai](https://www.nightai.co/). Recently, there have been many study groups like this online and they are great ways to immerse in the world of ML.
\\
***The landscape of Reinforcement Learning***
\\
Learn from Dr. Katja Hofmann the key concepts and methods of reinforcement learning and where it’s headed in this [webinar series](https://note.microsoft.com/MSR-Webinar-RL-Algorithm-to-Adoption-Registration-Live.html?wt.mc_id=twitter_MSR-WBNR_post_v3).
# Notable Mentions ⭐️
Check out t[his clean and self-contained PyTorch implementation](https://gist.github.com/y0ast/d91d09565462125a1eb75acc65da1469) of ResNet-18 applied to CIFAR-10 which achieves ~94% accuracy.
\\
PyTorch 1.4 is released! Check out release notes [here](https://github.com/pytorch/pytorch/releases/tag/v1.4.0).
\\
Elona Shatri wrote this excellent [summary](https://medium.com/@e.shatri1/what-is-optical-music-recognition-6515d8a53e01) about how she intends to approach optical music recognition using deep learning.
\\
The title for this blog post is self-explanatory: “T[he Case for Bayesian Deep Learning](https://cims.nyu.edu/~andrewgw/caseforbdl/)”.
\\
Chris Said shares his [experience](https://chris-said.io/2020/01/10/optimizing-sample-sizes-in-ab-testing-part-I/) in optimizing sample sizes for A/B testing, an important part of practical data science. Topics include the costs and benefits of large sample sizes and best practices for practitioners.
\\
Neural Data Server (NDS) is a dedicated search engine for obtaining transfer learning data. Read about the method [here](https://arxiv.org/abs/2001.02799) and the service [here](http://aidemos.cs.toronto.edu/nds/).
---
## File: posts/_posts/2020-01-19-NLP简报_Reformer,_DeepMath,_ELECTRA,_TinyBERT.md
---
layout: post
title: "NLP简报 #2 [ZH]: Reformer, DeepMath, ELECTRA, TinyBERT for Search, VizSeq, Open-Sourcing ML,…"
author: billy_rick
modified:
comments: true
excerpt: ""
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_2.png
---
\\
*Thanks to* [*Kaiyuan*](https://blog.csdn.net/Kaiyuan_sjtu) *(WeChat: NewBeeNLP) for this great translation.*
\\
欢迎回到 NLP 简报第二期！**如果想让自己有趣的研究/项目出现在 NLP 简报中，随时在公众号后台留言联系我**
\\
来看看本期的内容，
- **1、Publications 📙**
- 1.1 关于信任模型的不确定性
- 1.2 Systematic generalization
- 1.3 Reformer
- 1.4 用于文本分类的无监督域自适应
- 1.5 改进的上下文表示
- 1.6 模型可解释性
- **2、Creativity and Society 🎨**
- 2.1 情感不匹配
- 2.2 多巴胺的理解和蛋白质折叠
- 2.3 有关ML的访谈
- **3、Tools and Datasets ⚙️**
- 3.1 生产环境中的PyTorch模型
- 3.2 可视化文本生成序列
- 3.3 效果最优的在线语音识别
- **4、Ethics in AI 🚨**
- 4.1 AI implications
- 4.2 现代自然语言处理的环境成本
- **5、Articles and Blog posts ✍️**
- 5.1 开源ML
- 5.2 计算机视觉的自监督学习
- 5.3 用于搜索的TinyBERT
- 5.4 主动转移学习
- 5.5 Revealing the Dark Secrets of BERT
- **6、Education 🎓**
- 6.1 Neural Nets for NLP
- 6.2 深度学习数学
- 6.3 Python课程和教程
- 6.4 Deep Learning State of the Art
- 6.5 在线学习和研究
- 6.6 The landscape of Reinforcement Learning
- **7、Noteworthy Mentions ⭐️**
# 1、Publications 📙
**1.1 关于信任模型的不确定性**
\\
来自 Google AI 的最新论文Can You Trust Your Model's Uncertainty?[1]发表在 NeurIPS 上，研究的是模型的概率是否反映了其预测分布失调和数据移位的能力。发现深度继承模型在数据集移位上表现更好（即改进了模型不确定性），而其他模型并没有随着数据集移位变得越来越不确定，而是确信地变成了错误。可以查看官方博客[2]了解更多。
\\
\\
**1.2 Systematic generalization**
\\
ICLR 上发表的一篇有趣的工作：提出了模块模型和通用模型之间的比较，以及它们在语言理解中进行系统概括的有效性。基于对视觉问题回答任务进行的推理评估，作者得出结论，可能需要明确的正则化函数和先验知识才能实现系统的概括。
\\
**1.3 Reformer**
\\
众所周知，由于在 attention layer 中执行昂贵的计算，因此 Transformer 模型在可以覆盖的上下文窗口中受到很大限制。因此，可能仅可能将 Transformer 模型应用于有限的文本大小或生成简短的语句或音乐。GoogleAI 最近发布了一种有效的 Transformer 模型变体，称为Reformer[4]。该方法的主要焦点是能够处理更高的上下文窗口，同时减少计算需求并提高内存效率。重整器使用局部敏感哈希（LSH）将相似的向量分组在一起，并从其中创建片段，从而实现并行处理。然后将注意力转移到这些较小的部分和相应的相邻部分上，这就是减少计算负荷的原因。使用可逆层可实现存储效率，可逆层允许在通过反向传播进行训练时按需重新计算每层的输入信息。这是一种简单的技术，避免了需要在内存中存储激活的模型。查看此Colab notebook[5]，以了解如何将 Reformer 模型应用于图像生成任务。
\\
\\
**1.4 用于文本分类的无监督域自适应**
\\
这项工作，Self-Attention with Relative Position Representations[6]，提出了将距离度量结合到附加损失函数中的方法，以训练模型并改善无监督域自适应。该模型被扩展为 DistanceNet Bandit 模型，该模型优化了“转移到低资源目标域”的结果。用这种方法解决的关键问题是如何处理来自不同域的数据之间的差异，特别是因为它涉及 NLP 任务（例如情感分析）。
\\
**1.5 改进的上下文表示**
\\
ELECTRA[7]提出了一种称为 token-detection 的样本效率更高的预训练任务，用于训练比掩盖的语言建模预训练方法（例如 BERT）更有效的语言模型。该模型 ELECTRA，在相同的数据和模型大小下，其上下文表示优于 BERT 和 XLNET。该方法特别适用于低计算方案， 这对建立更小和更便宜的语言模型又迈进了一步。
\\
**1.5 模型可解释性**
\\
Distill 最近出版的标题为“Visualizing the Impact of Feature Attribution Baselines[8]”的文章讨论了Integrated Gradients[9]，该梯度用于通过识别哪些特征与预测某个数据点相关来解释各种问题中的神经网络。问题在于正确定义和保留缺失概念，这是积分梯度的基线输入所要达到的目的。在模型可解释性的背景下，这里的挑战在于，该方法必须确保模型不会突出显示缺失的特征，而同时又避免给基线输入以零重要性，而这很容易发生。作者建议定量评估一些先前使用和建议的基线选择的不同效果，以更好地保留缺失概念。
## **2、Creativity and Society 🎨**
**2.1 情感不匹配**
\\
这项纵向研究，Emotions Extracted from Text vs. True Emotions[10]，发现通过使用基于文本的算法提取的情绪通常与自我报告的情绪不同。
\\
**2.2 多巴胺的理解和蛋白质折叠**
\\
DeepMind 最近在 Nature 杂志上发表了两篇有趣的论文。第一篇论文，Dopamine-and-temporal-difference-learning[11]，旨在通过强化学习更好地了解大脑中的多巴胺如何发挥作用。第二篇论文，AlphaFold-Using-AI-for-scientific-discovery[12]，与蛋白质折叠[13]更相关，并试图更好地理解它，以便能够潜在地发现多种疾病的治疗方法。这些都是很好的例子，说明了如何将 AI 系统潜在地应用于现实世界中的应用程序以帮助社会。
\\
\\
**2.3 有关 ML 的访谈**
\\
在接受 Wired 采访时，Refik Anadol 讨论了机器学习算法创造精美艺术[14]的潜力。这是如何将 ML 用于创造力的一个很好的例子。
\\
人工智能可能会产生重大影响的行业之一是教育。在The Future of Everything[15]的新剧集中，Russ Altman 和 Emma Brunskill 对计算机辅助学习进行了深入讨论。
# 3、Tools and Datasets ⚙️
**3.1 生产环境中的 PyTorch 模型**
\\
Cortex 是一种工具，可用于自动化基础架构并将 PyTorch 模型作为 API 部署在 AWS 中。点击这篇博文Using PyTorch Models in Production with Cortex[16]详细了解其操作方式。
\\
**3.2 可视化文本生成序列**
\\
Facebook AI 发布了VizSeq[17]，该工具可帮助以可视化方式评估 BLUE 和 METEOR 等指标下的文本生成序列。该工具的主要目标是通过利用可视化并使其对研究人员更具可扩展性和生产力，提供对文本数据集的更直观的分析。可以阅读原论文[18]获取更多信息。
\\
\\
**3.3 效果最优的在线语音识别**
\\
FacebookAI 开源wav2letter@anywhere[19]，这是一个推理框架，该推理框架基于基于 Transformer 的声学模型，用于最新的在线语音识别。重大改进涉及模型的大小，并减少了音频和转录之间的延迟，这对于实现更快的实时推理都很重要。
\\
## **4、Ethics in AI 🚨**
**4.1 AI implications**
\\
为了防止 AI 系统对公众的滥用和不道德行为，欧盟正在考虑五年内禁止公众使用面部识别技术，点此查看全文[20]。
\\
**4.2 现代自然语言处理的环境成本**
\\
也许大多数时候都被忽略了，Energy and Policy Considerations for Deep Learning in NLP 一文[21]讨论了 NLP 中现代深度学习方法的能量和政策考虑。众所周知，当前的模型依赖数十亿个参数，进而依赖大量的计算资源，从而消耗大量能源。作者希望对训练这些现代 NLP 模型所涉及的环境成本传播更多的认识。
\\
Zachary Lipton[22]在多伦多大学的演讲中讨论了公平性，可解释性和解决主义的危险，演讲主题围绕 ML 公平方法和含义。
# 5、Articles and Blog posts ✍️
**5.1 开源 ML**
\\
Hugging Face 的科学负责人 Thomas Wolf 为计划开源 ML 代码&研究[23]的人员提供了出色的建议。
\\
**5.2 计算机视觉的自监督学习**
\\
Jeremy Howard 在这篇出色的博客文章中，Self-supervised learning and computer vision[24]，简要介绍了在计算机视觉环境下进行自监督学习的概念。这些简短的摘要，有助于为你提供可靠的介绍，万一你有兴趣将这一领域的技术应用于自己的问题呢。
\\
**5.3 用于搜索的 TinyBERT**
\\
我们已经看到许多 BERT 模型的变体（例如 DistilBERT）的成功，这些变体使用某种形式的知识蒸馏来显着减小模型大小并提高速度。一些人使用了 BERT 的变体 TinyBERT，并将其应用于基于关键字的搜索解决方案。此项目的灵感来自于该搜索解决方案，该解决方案用于理解 Google 提出的搜索。该体系结构的很大一部分是，它可以在标准 CPU 上工作，并且可以用于改善和理解搜索结果。
\\
**5.4 主动转移学习**
\\
Rober Monarch 这篇有关主动转移学习[25]的博客文章十分有趣，这是他即将出版的《Human-in-the-loop Machine Learning[26]》一书的一部分。他目前正在写很棒的博客文章，介绍结合人与机器智能解决问题的方法，他还提供了所讨论方法的随附 PyTorch 实现。
\\
**5.5 Revealing the Dark Secrets of BERT**
\\
Anna Roger 撰写了一篇有趣而有趣的博客文章， the Dark Secrets of BERT[27]，其中讨论了经过微调的 BERT 的实际情况，以及所宣称的优势是否用于处理诸如情感分析，文本含义和自然语言推断等下游任务。分析的结果表明，BERT 的参数过高，并且该框架的自注意力组件由于它与被编码并用于推理的语言信息有关，所识别出的好处不一定特别如其所主张的。
# 6、Education 🎓
**6.1 Neural Nets for NLP**
\\
CMU 的 NLP 教授 Graham Neubig 已经发布了本学期提供的“ NLP 神经网络”课程[28]的视频。对于那些对现代 NLP 方法学习感兴趣的人，我强烈推荐该播放列表。
\\
**6.2 深度学习数学**
\\
是否想深入研究深度学习方法背后的数学[29]？这个视频讲座系列，已经拥有众多演讲者。
\\
**6.3 Python 课程和教程**
\\
Python 不仅在 IT 行业而且在数据科学领域也已成为最受欢迎的编程语言之一。为了向全世界的学习者提供 Python 的实践知识，Google 发布了“Google IT Automation with Python Professional Certificate[30]”课程。尽管该课程与 ML 或 AI 并没有直接关系，但这绝对是精通 Python 语言的不错的基础课程，并且提供奖学金。
\\
这是另一个很有前景的视频系列，称为“Deep Learning (for Audio) with Python[31]”，重点是利用 Tensorflow 和 Python 通过利用深度学习来构建与音频/音乐相关的应用程序。
\\
\\
**6.4 Deep Learning State of the Art**
\\
观看 Lex Fridman 关于深度学习的最新研究和发展[32]的视频讲座。他谈到了诸如感知器，神经网络，反向传播，CNN，深度学习，ImageNet，GAN，AlphaGo 和最新的 Transformers 等主题的重大突破，本讲座是 MIT 深度学习系列的一部分。
\\
**6.5 在线学习和研究**
\\
有许多很棒的在线计划可以在研究和学习中进行协作。我个人最喜欢的是MLT 的数学阅读课程[33]，这项由 Nightai 发起的新的分布式 AI 研究协作计划。最近，有许多这样的在线研究小组，它们是沉浸于 ML 世界的好方法。
\\
**6.6 The landscape of Reinforcement Learning**
\\
在网络研讨会上向 Katja Hofmann 博士学习强化学习[34]的关键概念、方法，以及学习方向。
# 7、Notable Mentions ⭐️
查看适用于 CIFAR-10 的ResNet-18 的 PyTorch 实现[35]，该实现可实现约 94％的准确度。
\\
PyTorch 1.4[36]发布了！在此处查看发行说明。
\\
Elona Shatri 撰写了一篇出色的摘要，说明了她打算如何使用深度学习进行光学音乐识别[37]。
\\
看博客文章的标题就知道啦：“贝叶斯深度学习案例[38]”。
\\
Chris Said 分享了他在优化 A/B 测试样本量[39]方面的经验，这是实用数据科学的重要组成部分。主题包括大样本量的成本和收益以及从业人员的最佳实践。
\\
Neural Data Server (NDS)[40] 是用于获取转移学习数据的专用搜索引擎， 在此处阅读有关方法和服务[41]的信息。
\\
**本文参考资料**
\\
[1] **Can You Trust Your Model's Uncertainty?:** https://arxiv.org/abs/1906.02530
\\
[2] **官方博客:** https://ai.googleblog.com/2020/01/can-you-trust-your-models-uncertainty.html
\\
[3] **Systematic Generalization: What Is Required and Can It Be Learned?:** https://www.semanticscholar.org/paper/Systematic-Generalization:-What-Is-Required-and-Can-Bahdanau-Murty/6c7494a47cc5421a7b636c244e13586dc2dab007
\\
[4] **Reformer:** https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html
\\
[5] **Colab notebook:** https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/image_generation.ipynb
\\
[6] **Self-Attention with Relative Position Representations:** https://arxiv.org/abs/2001.04362
\\
[7] **ELECTRA:** https://openreview.net/forum?id=r1xMH1BtvB
\\
[8] **Visualizing the Impact of Feature Attribution Baselines:** https://distill.pub/2020/attribution-baselines/
\\
[9] **Integrated Gradients:** https://medium.com/@kartikeyabhardwaj98/integrated-gradients-for-deep-neural-networks-c114e3968eae
\\
[10] **Emotions Extracted from Text vs. True Emotions:** https://ieeexplore.ieee.org/abstract/document/8952437
\\
[11] **Dopamine-and-temporal-difference-learning:** https://deepmind.com/blog/article/Dopamine-and-temporal-difference-learning-A-fruitful-relationship-between-neuroscience-and-AI
\\
[12] **AlphaFold-Using-AI-for-scientific-discovery:** https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery[13]**蛋白质折叠:** https://en.wikipedia.org/wiki/Protein_folding
\\
[14] **机器学习算法创造精美艺术:** https://www.youtube.com/watch?v=I-EIVlHvHRM&feature=youtu.be
\\
[15] **The Future of Everything:** https://engineering.stanford.edu/magazine/article/emma-brunskill-amped-education-ai?sf115875862=1
\\
[16] **Using PyTorch Models in Production with Cortex:** https://medium.com/pytorch/how-to-build-production-software-with-pytorch-9a8725382f2a
\\
[17] **VizSeq:** https://ai.facebook.com/blog/vizseq-a-visual-analysis-toolkit-for-accelerating-text-generation-research/
\\
[18] **论文:** https://www.aclweb.org/anthology/D19-3043.pdf
\\
[19] **wav2letter@anywhere:** https://ai.facebook.com/blog/online-speech-recognition-with-wav2letteranywhere/
\\
[20] **查看全文:** https://www.reuters.com/article/us-eu-ai/eu-mulls-five-year-ban-on-facial-recognition-tech-in-public-areas-idUSKBN1ZF2QL
\\
[21] **Energy and Policy Considerations for Deep Learning in NLP一文:** https://arxiv.org/abs/1906.02243
\\
[22] **Zachary Lipton:** https://c4ejournal.net/2020/01/16/zack-lipton-fairness-interpretability-and-the-dangers-of-solutionism-ethics-of-ai-in-context2020-c4ej-2/
\\
[23] **开源ML代码&研究:** https://twitter.com/Thom_Wolf/status/1216990543533821952?s=20
\\
[24] **Self-supervised learning and computer vision:** https://www.fast.ai/2020/01/13/self_supervised/
\\
[25] **主动转移学习:** https://medium.com/pytorch/https-medium-com-robert-munro-active-learning-with-pytorch-2f3ee8ebec
\\
[26] **Human-in-the-loop Machine Learning:** https://www.manning.com/books/human-in-the-loop-machine-learning
\\
[27] **the Dark Secrets of BERT:** https://text-machine-lab.github.io/blog/2020/bert-secrets/
\\
[28] **“ NLP神经网络”课程:** https://www.youtube.com/playlist?list=PL8PYTP1V4I8CJ7nMxMC8aXv8WqKYwj-aJ
\\
[29] **深度学习方法背后的数学:** https://www.youtube.com/playlist?list=PLWQvhvMdDChzsThHFe4lYAff3pu2m0v2H
\\
[30] **Google IT Automation with Python Professional Certificate:** https://www.coursera.org/professional-certificates/google-it-automation
\\
[31] **Deep Learning (for Audio) with Python:** https://www.youtube.com/watch?v=fMqL5vckiU0&list=PL-wATfeyAMNrtbkCNsLcpoAyBBRJZVlnf
\\
[32] **深度学习的最新研究和发展:** https://www.youtube.com/watch?v=0VH1Lim8gL8
\\
[33] **MLT的数学阅读课程:** https://twitter.com/**「MLT」**
\\
[34] **强化学习:** https://note.microsoft.com/MSR-Webinar-RL-Algorithm-to-Adoption-Registration-Live.html?wt.mc_id=twitter_MSR-WBNR_post_v3
\\
[35] **ResNet-18的PyTorch实现:** https://gist.github.com/y0ast/d91d09565462125a1eb75acc65da1469
\\
[36] **PyTorch 1.4:** https://github.com/pytorch/pytorch/releases/tag/v1.4.0
\\
[37] **使用深度学习进行光学音乐识别:** https://medium.com/@e.shatri1/what-is-optical-music-recognition-6515d8a53e01
\\
[38] **贝叶斯深度学习案例:** https://cims.nyu.edu/~andrewgw/caseforbdl/
\\
[39] **优化A/B测试样本量:** https://chris-said.io/2020/01/10/optimizing-sample-sizes-in-ab-testing-part-I/
\\
[40] **Neural Data Server (NDS):** https://arxiv.org/abs/2001.02799[41]**服务:** http://aidemos.cs.toronto.edu/nds/
---
## File: posts/_posts/2020-02-01-NLP_Newsletter_Flax,_Thinc,_Language-specific_BERT.md
---
layout: post
title: "NLP Newsletter #3 [EN]: Flax, Thinc, Language-specific BERT models, Meena, Flyte, LaserTagger,…"
author: billy_rick
modified:
comments: true
excerpt: "The third issue covers topics such as improving conversational agents, releases of language-specific BERT models, free datasets, releases of deep learning libraries, and much more."
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_3.png
---
\\
Welcome to the NLP Newsletter! The third issue covers topics such as improving conversational agents, releases of language-specific BERT models, free datasets, releases of deep learning libraries, and much more.
\\
Don’t forget to subscribe to the publication or follow us on [Twitter](https://twitter.com/dair_ai) 🐦 to get the latest of the NLP Newsletter and dair.ai.
# Publications 📙
***Language-specific BERT models***
\\
I have lost count of how many language-specific BERT models there are now but here a couple of the recent releases:
- Dutch BERT ([RobBERT](https://arxiv.org/abs/2001.06286), [BERTje](https://arxiv.org/abs/1912.09582))
- [German BERT](https://deepset.ai/german-bert)
- [Portuguese BERT](https://github.com/neuralmind-ai/portuguese-bert)
- French BERT ([CamemBERT](https://arxiv.org/abs/1911.03894), [FlauBERT](https://arxiv.org/abs/1912.05372))
- Italian BERT ([AlBERTo](http://ceur-ws.org/Vol-2481/paper57.pdf), [UmBERTo](https://github.com/musixmatchresearch/umberto))
- Spanish BERT ([BETO](https://github.com/dccuchile/beto))
- Arabic BERT ([araBERT](https://colab.research.google.com/drive/1KSy89fAkWt6EGfnFQElDjXrBror9lIZh))
\\
Note that most of these models are also being made available through the Hugging Face’s [Transformer library](https://huggingface.co/models), which was recently upgraded to [2.4.1](https://github.com/huggingface/transformers/releases).
\\
***Overly Optimistic Prediction Results on Imbalanced Data: Flaws and Benefits of Applying Over-sampling***
\\
This [paper](https://arxiv.org/abs/2001.06296) reveals and extensively discusses some of the flaws and benefits of applying over-sampling to deal with imbalanced datasets before partitioning the datasets. Furthermore, the work reproduces previous studies and identifies this methodological flaw that produces overly optimistic results.
\\
***Encode, Tag and Realize: A Controllable and Efficient Approach for Text Generation***
\\
In an effort to reduce the effect of [hallucination](https://arxiv.org/abs/1910.08684) (producing outputs not supported by input text) common in seq2seq-based text generation methods, a group of Google engineers open-sourced a method for text generation called [LaserTagger](https://ai.googleblog.com/2020/01/encode-tag-and-realize-controllable-and.html). The main idea of the method is to produce outputs by tagging words with predicted edit operations (e.g., KEEP, DELETE-ADD, etc.) and applying it to the input words in a so-called *realization* step. This replaces the common text generation method that just generates output from scratch which is generally slow and error-prone. The model offers other benefits besides generating fewer errors, such as enabling the parallel predictions of edit operations while still maintaining good accuracy and outperforming a BERT baseline in scenarios with a smaller number of training examples.
\\
*“LaserTagger applied to sentence fusion. The predicted edit operations correspond to deleting “. Turing” and adding “and he” before it. Notice the high overlap between the input and output text.” —* [*source*](https://ai.googleblog.com/2020/01/encode-tag-and-realize-controllable-and.html)
\\
***Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future***
\\
Grace Lindsay released this nice and easy-to-read [report](https://arxiv.org/abs/2001.07092) on the history of CNNs and how they are evaluated as models of biological vision, i.e., how do CNN representations compare to those of the brain? The discussion of the emerging opportunities on the use of CNNs for vision research is highly recommended for readers.
\\
[*source*](https://arxiv.org/abs/2001.07092)
\\
***Multilingual Denoising Pre-training for Neural Machine Translation***
\\
Facebook AI published [mBART](https://arxiv.org/pdf/2001.08210.pdf) a method based on a multilingual seq2seq denoising auto-encoder pretrained on large-scale monolingual corpora for machine translation across 25 languages. This work follows the pretraining scheme of Lewis et al., 2019, BART, and investigates the effects of denoising pretraining on multiple languages such as Korean, Japanese, and Turkish, among other languages. Input text involves the masking of phrases and permuting sentences (noising). A Transformer-based model is learned to reconstruct the text across many languages. The complete autoregressive model is trained only once and can be fine-tuned on any of the language pairs without involving any task-specific or language-specific modifications. Both document-level and sentence-level translation problems are approached. Besides showing performance gains, the authors claim that the method works well on low-resource machine translation.
\\
[*source*](https://arxiv.org/pdf/2001.08210.pdf)
\\
***On improving conversational agents***
\\
[Meena](https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html) is a neural conversational agent that aims to conduct improved conversations that are more sensible and specific — metrics that are defined to capture important attributes from a human conversation (e.g., fluency). The model learns the conversation context via an encoder and formulates a *sensible* response via the decoder. It is reported that improvement in the quality of conversations was possible by considering more powerful decoders.
\\
*You can also catch Alan Nichol’s (Co-Founder of Rasa HQ)* [*thoughts*](https://venturebeat.com/2020/01/31/with-googles-meena-are-ai-assistants-about-to-get-a-lot-smarter/) *on this work.*
# Creativity and Society 🎨
***ML tools — reading comprehension test and sentiment analyzer***
\\
Ming Cheuk built this [fun app](https://littlealbert.now.sh/#/) that allows you to test the reading comprehension capabilities of a model trained on deep learning methods, in particular, using Google AI’s language model called [ALBERT](https://ai.googleblog.com/2019/12/albert-lite-bert-for-self-supervised.html). ALBERT is a tiny version of BERT for learning language representations. The author explains more about the project and approaches used in this [blog post](https://www.spark64.com/post/machine-comprehension).
Hendrik Strobelt also released a small [project](https://github.com/HendrikStrobelt/sentimenter_minimal_hai) where he shows how to prototype an interactive sentiment analyzer. These are all projects that help to highlight some of the great ideas coming from ML and NLP. If you have a serious and interesting project you are working on, please reach out and we can feature it in the publication/newsletter.
\\
[*source*](https://littlealbert.now.sh/#/)
\\
***The journey of a self-taught AI researcher at Google***
\\
In this candid [interview](https://blog.floydhub.com/emils-story-as-a-self-taught-ai-researcher/), you can hear directly from Emil, an ML researcher at Google Art & Culture, about his journey in the pursuit of a career in AI as a self-taught researcher.
# Tools and Datasets ⚙️
***Free Datasets***
\\
[Google Dataset Search](https://blog.google/products/search/discovering-millions-datasets-web/) is officially out of beta and now provides up to 25 million datasets to search upon. If you ever want inspiration for your next data science or machine learning project, this is the place to find references to datasets hosted all over the internet. It’s basically a search engine for datasets, this is amazing work and required tremendous effort!
\\
The [Big Bad NLP Database](https://quantumstat.com/dataset/dataset.html) is a website where you can search a dedicated database of over 200 NLP datasets of all types for tasks such as commonsense, sentiment analysis, questions answering, entailment inference, among others.
\\
\\
***Reinforcement learning library***
\\
There are several good reinforcement libraries out there. Recently, Chris Nota developed and released a [PyTorch library](https://github.com/cpnota/autonomous-learning-library) for building reinforcement learning agents based on popular deep RL algorithms such as DQN, PPO, and DDPG, among others. The emphasis of the library is on object-oriented design and for enabling the quick implementation and evaluation of novel reinforcement learning agents.
\\
***ML Explainability and Interpretability***
\\
If you are currently working with text-based language models and want to understand how to interpret them more easily when applied to different language tasks, then you may be interested in [Captum](https://captum.ai/). Captum is an interpretability library that can be used to analyze feature importance, interpreting text and vision models, interpreting multimodal models, and other models such as BERT used for question answering.
\\
If you are interested in model explainability, this [set of tutorials](https://www.kaggle.com/learn/machine-learning-explainability) may also interest you. It includes methods for understanding feature importance via notebooks.
\\
***Machine learning and deep learning libraries***
\\
The Google Research team released [Flax](https://github.com/google-research/flax/tree/prerelease) — a flexible and powerful neural network library based on [JAX](https://github.com/google/jax) which provides a framework for fast computing and training machine learning models using the typical Numpy APIs.
\\
*Flax syntax*
\\
[Thinc](https://thinc.ai/) is a lightweight deep learning library developed by the makers of spaCy. It offers functional-programming APIs for composing, configuring, and deploying custom models built with libraries like PyTorch and TensorFlow.
\\
Lyft releases [Flyte](https://eng.lyft.com/introducing-flyte-cloud-native-machine-learning-and-data-processing-platform-fb2bb3046a59) which is a multi-tenant, production-ready, and serverless platform for deploying concurrent, scalable, and maintainable ML and data processing workflows.
\\
***A tool for conversational AI***
\\
The open-source conversational AI framework [DeepPavlov](https://github.com/deepmipt/DeepPavlov) offers a free and easy-to-use solution for building dialogue systems and complex conversational systems. DeepPavlov comes with several predefined components for solving NLP-related problems. It integrates BERT (including conversational BERT) into three downstream tasks: text classification, named entity recognition (and sequence tagging in general), and question answering. As a result, it achieved substantial improvements in all these tasks. ([Google Colab](https://colab.research.google.com/github/deepmipt/dp_notebooks/blob/master/DP_tf.ipynb) | [Blog](https://medium.com/tensorflow/deeppavlov-an-open-source-library-for-end-to-end-dialog-systems-and-chatbots-31cf26849e37) | [Demo](https://demo.deeppavlov.ai/#/en/textqa))
# Ethics in AI 🚨
***Facial recognition and privacy***
\\
The New York Times wrote an interesting report on different perspectives of privacy as it relates to facial recognition technology. The focus of this story is around a “secretive company” called Clearview which allegedly uses AI technology to build universal facial recognition using images scraped from social media sites such as Twitter, Facebook, and YouTube, among others. The said technology raises concerns about privacy but it is also claimed to be used mostly for law enforcement. Read more of the story [here](https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html).
\\
***Human-Level AI Progress***
\\
In this [report](https://fortune.com/longform/ai-artificial-intelligence-big-tech-microsoft-alphabet-openai/), Jeremy Kahn extensively discusses the difference between “Narrow AI” and “General AI” in the context of the current progress of AI technology. Besides the many topics discussed, lots of questions arise on the returns of (if ever possible) achieving AGI. The report also mentions the recent interest of big tech companies investing in these efforts. Most notably, the report includes several concerns raised by respected researchers claiming the “irresponsible and unethical” behavior displayed by some research organizations that try to manipulate AI narratives for their benefit.
\\
***Privacy-preserving AI technologies***
\\
I am a strong believer in ethical and responsible AI, it’s not a fad nor is it the time to downplay it, it’s a serious endeavor that requires the best of minds to get it right. One of the biggest efforts I have witnessed in this aspect of AI is the privacy-persevering efforts from [OpenMined](https://twitter.com/OpenMinedOrg), a community that aims to build more responsible AI tools with privacy as a first-class citizen. If you want to know more, listen to Andrew Trask, lead of this effort, talk more about this initiative in this [recording](https://www.youtube.com/watch?v=4zrU54VIK6k) that is part of the MIT Deep Learning lecture series.
\\
***Understanding AI Ethics and Safety***
\\
Dr. David Leslie published this very [detailed report](https://www.turing.ac.uk/sites/default/files/2019-06/understanding_artificial_intelligence_ethics_and_safety.pdf) on topics that help to better understand AI in the context of ethic and safety. It aims to help developers and researchers better design and implement AI systems for the public sector.
\\
[*source*](https://www.turing.ac.uk/sites/default/files/2019-06/understanding_artificial_intelligence_ethics_and_safety.pdf)
# Articles and Blog posts ✍️
***Speeding up tokenization tutorial***
\\
Steven van de Graaf wrote this [article](https://towardsdatascience.com/a-small-timing-experiment-on-the-new-tokenizers-library-a-write-up-7caab6f80ea6) reporting performance gains when using Hugging Face’s new [Tokenizers](https://github.com/huggingface/tokenizers) library as compared to the standard, built-in tokenizer used in [Transformers](https://github.com/huggingface/transformers). Steven reports 9x speed-up and an implementation that takes 10.6 seconds to tokenize 1 million sentences.
\\
***Can language models really comprehend?***
\\
The Gradient recently featured this [post](https://thegradient.pub/gpt2-and-the-nature-of-intelligence/) by Gary Marcus where he discusses what he believes are fundamental flaws behind language models like GPT-2. The main argument of Gary Marcus is that a model that is trained to be able to predict the next word is not necessarily a model that could understand or reason, i.e., “Prediction is a component of comprehension, not the whole thing.” He also discusses the importance of innateness in the context of language and argues that the current language models don’t consider it.
\\
***Curriculum for Reinforcement Learning***
\\
Can designing curriculum-based methods help an RL agent to learn? Lillian Weng [summarizes](https://lilianweng.github.io/lil-log/2020/01/29/curriculum-for-reinforcement-learning.html) several curriculum-based approaches and how they can be leveraged to train effective reinforcement learning agents. Weng discusses the challenges of designing an efficient curriculum learning approach which generally requires sorting complexity of tasks and providing the model a sequence of tasks that increase in the level of difficulty during training.
\\
[*source*](https://lilianweng.github.io/lil-log/2020/01/29/curriculum-for-reinforcement-learning.html)
\\
***Introduction to NumPy***
\\
I always recommend anyone that is getting started with machine learning to get plenty of practice with NumPy. A lot of the high-level libraries used for deep learning and machine learning today use NumPy APIs in some way so it’s a very important tool to understand internally. Anne Bonner recently released this very detailed [tutorial](https://numpy.org/devdocs/user/absolute_beginners.html) introducing the basics of NumPy.
\\
[*source*](https://numpy.org/devdocs/user/absolute_beginners.html)
# Education 🎓
***Foundations of machine learning and statistical inference***
\\
Anima Anandkumar, from Caltech, released a course titled “**Foundations of machine learning and statistical inference**”. The course focuses on ML concepts such as matrices, tensors, optimization, probabilistic models, neural networks and much more. This is a great course because it focuses on the theoretical aspect of ML, which is just as essential to understand and improve more advanced methods. ([video playlist](https://www.youtube.com/playlist?list=PLVNifWxslHCDlbyitaLLYBOAEPbmF1AHg) | [course syllabus](http://tensorlab.cms.caltech.edu/users/anima/cms165-2020.html))
\\
***Deep Learning Lecture Series***
\\
DeepMind partnered with UCL to release the [Deep Learning Lecture series](https://www.eventbrite.co.uk/o/ucl-x-deepmind-deep-learning-lecture-series-general-29078980901), which includes 12 lectures that will be delivered by leading research scientists at DeepMind. Topics include how to train a neural network using methods such as attention, memory, and generative modeling, and much more.
\\
***~7 million syllabi***
\\
Education is a vital part of growing communities and entire industries. It is where the seed of innovation is planted. [Open Syllabus](https://opensyllabus.org/) is a non-profit organization that is using the power of crowdsourcing to map the curriculum of higher education into one free-for-all online database. It currently contains roughly seven million syllabi.
\\
[*source*](https://opensyllabus.org/results-list/titles?size=50&fields=Computer%20Science)
\\
***Discussing, Sharing, and Learning about ML***
\\
[r/ResearchML](https://www.reddit.com/r/ResearchML/) is a new machine learning subreddit for discussing ML papers. This one is more focused on research and encouraging deeper discussions.
\\
[practicalAI](https://practicalai.me/explore/content/) is a website where you can explore and discover ML topics curated by the community and experts.
\\
# Notable Mentions ⭐️
\\
[Learn](https://github.blog/2020-01-22-how-we-built-good-first-issues/) more about how GitHub is leveraging machine learning to spot easy and personalized issues for developers so that they can tackle issues that are aligned with their interests. This encourages quicker and more contributions from open-source contributors.
\\
Catch up with the latest top NLP news on the [recent issue](http://newsletter.ruder.io/issues/nlp-progress-restrospectives-and-look-ahead-new-nlp-courses-independent-research-initiatives-interviews-lots-of-resources-217744) of the NLP News by Sebastian Ruder. Highlights include an update of NLP Progress, retrospectives of the past decade, new NLP courses, among other topics.
\\
Check out this incredible list of TensorFlow 2.0 [deep learning notebooks](https://github.com/NERSC/dl4sci-tf-tutorials) that range from CycleGAN to Transformers to image captioning tasks. They were publicly released by the Deep Learning for Science School at LBNL.
\\
An impressive and easy to follow blog [post](https://engineering.papercup.com/posts/bayesian-neural-nets/) explaining the basics of Bayesian neural networks. A great introduction for starters.
\\
John Schulman [shares](http://joschu.net/blog/opinionated-guide-ml-research.html) some advice for upcoming ML researchers on how to better choose research problems and be more strategic about implementing and tackling the research tasks at hand. John also shares tips for personal development and continual progress.
----------
**Editor’s note:** Thanks for all the love ❤️ and support 🤗 you have been giving to the newsletter and this publication. It helps to remind me of how important this effort is and why I should always take the time to distill the best of ML and NLP that’s happening more recently. If you would love to contribute 💪, please feel free to reach out. Suggestions are highly welcome, this is a community effort as I see it. Also, the official website of [dair.ai](https://dair.ai/) is ready! I will make the official announcement here in an upcoming post. The website aims to simplify and extend the reach to readers from all over the world 🌐.
---
## File: posts/_posts/2020-02-15-NLP简报_Flax,_Thinc,_Language-specific_BERT_models.md
---
layout: post
title: "NLP简报 #3 [ZH]: Flax, Thinc, Language-specific BERT models, Meena, Flyte, LaserTagger,…"
author: billy_rick
modified:
comments: true
excerpt: ""
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_3.png
---
\\
**Thanks to [Kaiyuan](https://blog.csdn.net/Kaiyuan_sjtu) (WeChat: NewBeeNLP) for this great translation.**
\\
欢迎来到 NLP 时事简报！涵盖了诸如特定语言 BERT 模型、免费数据集、深度学习库等主题。
# 1、Publications 📙
**1.1 Language-specific BERT models**
\\
我已经记不清现在有多少种特定语言的 BERT 模型了，这里有一些最新的版本:
- 荷兰语 Dutch BERT（RobBERT[1]| BERTje[2]）
- 德语 German BERT[3]
- 葡萄牙语 Portuguese BERT[4]
- 法语（CamemBERT[5] | FlauBERT[6]）
- 意大利语（AlBERTo[7] | UmBERTo[8]）
- 西班牙语（BETO[9]）
- 阿拉伯语（araBERT[10]）
\\
大多数这些模型也可以通过 huggingFace 的Transformer 库[11]获得，该库最近升级到了2.4.1[12]。
\\
**1.2 Overly Optimistic Prediction Results on Imbalanced Data: Flaws and Benefits of Applying Over-sampling**
\\
这篇论文[13]揭示并广泛讨论了在对数据集进行划分之前应用过采样来处理不平衡数据集的缺点和优点。此外，该工作复现了先前的研究，并确定了导致过于乐观的结果的方法论缺陷。
\\
**1.3 Encode, Tag and Realize: A Controllable and Efficient Approach for Text Generation**
\\
为了减少基于 seq2seq 的文本生成方法中常见的 hallucination [14]（产生输入文本不支持的输出）的影响，Google 工程师公开了一种称为LaserTagger[15]的文本生成方法。该方法的主要思想是通过使用预测的编辑操作（例如`KEEP`，`DELETE-ADD`等）标记单词并在所谓的realization step中将其应用于输入单词来产生输出。
\\
这代替了通常只从头生成输出的文本生成方法，这种方法通常很慢而且容易出错。该模型除了产生更少的错误外，还提供了其他好处，例如，在进行少量训练示例的情况下，可以在实现并行预测的同时进行编辑操作，同时仍保持良好的准确性并优于 BERT baseline。
\\
\\
**1.4 Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future**
\\
Grace Lindsay 发布了这份关于 CNN 历史的精美且易于阅读的报告[16]，以及如何将它们作为生物视觉的模型进行评估，即 CNN 表示与大脑的表示相比如何？强烈建议读者关注这份利用 CNN 进行视觉研究的讨论。
\\
\\
**1.5 Multilingual Denoising Pre-training for Neural Machine Translation**
\\
Facebook AI 发布了mBART[17]，一种基于多语言 seq2seq 去噪自动编码器的方法，该方法在大规模单语言语料库上进行了预训练，可用于 25 种语言的机器翻译。这项工作遵循 Lewis 等人（BART，2019）的预训练方案，并研究了去噪预训练对韩文，日文和土耳其文等多种语言的影响。输入文本涉及对短语的掩盖和对句子的置换（加噪），学习了一种基于 Transformer 的模型跨多种语言重建文本。完整的自回归模型仅训练一次，并且可以在任何语言对上进行微调而无需进行任何特定于任务或特定于语言的修改。此外，解决了文档级和句子级的翻译问题。除了表现出性能提升外，作者还声称该方法在低资源机器翻译方面效果很好。
\\
\\
**1.6 On improving conversational agents**
\\
Meena[18]是一种 neural conversational agents，旨在进行更明智和更具体的改进对话 — — 定义为从人类对话中捕获重要属性（例如，流畅度）的指标。该模型通过编码器学习会话上下文，并通过解码器制定合理的响应。据报道，通过考虑使用更强大的解码器可以提高通话质量。
\\
你也可以了解更多 Alan Nichol（Rasa HQ 的联合创始人）关于这项工作的想法[19]。
# 2、Creativity and Society 🎨
**2.1 ML tools — reading comprehension test and sentiment analyzer**
\\
Ming Cheuk 构建了这个有趣的应用程序Albert Learns to Read[20]，可以测试经过深度学习方法训练的模型的阅读理解能力，尤其是使用 Google AI 的语言模型ALBERT[21]。ALBERT 是 BERT 的微型版本，用于学习语言表示。作者在博客 Teaching Machines to Read[22]详细介绍了该项目和使用的方法。
\\
\\
**2.2 A Self-Taught AI Researcher at Google**
\\
在这个采访A Self-Taught AI Researcher at Google[23]中，你可以直接从 Google Art＆Culture 的 ML 研究人员 Emil 那里听到有关他作为一名自学成才的研究人员从事 AI 事业的经历。
# 3、Tools and Datasets ⚙️
**3.1 Free Datasets**
\\
Google 数据集搜索[24]正式退出测试版，现在可提供多达 2500 万个数据集进行搜索。如果你想获得下一个数据科学或机器学习项目的灵感，那么这里是查找对整个 Internet 上托管的数据集的引用的地方。它基本上是用于数据集的搜索引擎，这是一项了不起的工作，需要付出巨大的努力！
\\
Big Bad NLP 数据库[25]是一个网站，你可以在其中搜索 200 多种 NLP 数据集的专用数据库，以执行诸如常识，情感分析，问题回答，蕴含推理等任务。
\\
\\
**3.2 Reinforcement learning library**
\\
最近，Chris Nota 开发并发布了PyTorch 库[26]，用于基于流行的深度 RL 算法（例如 DQN，PPO 和 DDPG 等）来构建强化学习代理。该库的重点是面向对象的设计，并能够快速实施和评估新型强化学习代理。
\\
**3.3 ML Explainability and Interpretability**
\\
如果你当前正在使用基于文本的语言模型，并且想了解在应用于不同语言任务时如何更轻松地解释它们，那么你可能会对Captum[27]感兴趣。Captum 是一个可解释性库，可用于分析功能重要性，解释文本和视觉模型，解释多峰模型以及其他模型（例如用于回答问题的 BERT）。
\\
如果你对模型的可解释性感兴趣，那么这套教程[28]也可能会让您感兴趣。它包括通过 notebook 了解功能重要性的方法。
\\
**3.4 Machine learning and deep learning libraries**
\\
Google Research 团队发布了Flax[29]，一种基于JAX[30]的灵活而强大的神经网络库，该库提供了使用典型的 Numpy API 进行快速计算和训练机器学习模型的框架。
\\
\\
Thinc[31]是由 spaCy 的开发者开发的轻量级深度学习库。它提供了功能编程 API，用于组成，配置和部署使用 PyTorch 和 TensorFlow 之类的库构建的自定义模型。
\\
Lyft 发布了Flyte[32]，它是一个多租户，可用于生产的无服务器平台，用于部署并发，可伸缩和可维护的 ML 和数据处理工作流。
\\
**3.5 A tool for conversational AI**
\\
开源对话式 AI 框架DeepPavlov[33]为构建对话系统和复杂的对话系统提供了免费且易于使用的解决方案。DeepPavlov 带有几个预定义的组件，用于解决与 NLP 相关的问题。它将 BERT（包括会话 BERT）集成到三个下游任务中：文本分类，命名实体识别（和一般的序列标记）以及问题解答。结果，它在所有这些任务上都取得了重大改进。(Google Colab[34] | Blog[35] | Demo[36])
# 4、Ethics in AI 🚨
**4.1 Facial recognition and privacy**
\\
纽约时报针对与面部识别技术有关的隐私的不同观点撰写了一篇有趣的报告。这个故事的重点是一个名为“ Clearview”的“秘密公司”，据称该公司使用 AI 技术通过从 Twitter，Facebook 和 YouTube 等社交媒体网站上抓取的图像来构建通用的面部识别。所述技术引起了人们对隐私的担忧，但是据称它还主要用于执法。点击此处[37]阅读更多故事。
\\
**4.2 Human-Level AI Progress**
\\
Jeremy Kahn 在这个报告[38]中广泛讨论了在 AI 技术的当前发展背景下“ Narrow AI”和“ General AI”之间的区别。除了讨论的许多主题之外，关于（如果可能的话）实现 AGI 的回报还有很多问题。该报告还提到了大型高科技公司最近对这些努力进行投资的兴趣。最值得注意的是，该报告包括一些受人尊敬的研究人员提出的一些担忧，他们声称某些试图操纵 AI 叙述以利于他们的研究组织表现出“不负责任和不道德”的行为。
\\
**4.3 Understanding AI Ethics and Safety**
\\
Dr.David Leslie 发表了这份非常详尽的报告[39]，主题涉及有助于在道德和安全方面更好地理解人工智能的主题。它旨在帮助开发人员和研究人员更好地为公共部门设计和实施 AI 系统。
\\
# 5、Articles and Blog posts ✍️
**5.1 Speeding up tokenization tutorial**
\\
Steven van de Graaf 撰写了这篇文章[40]，报告说，与使用 Transformers 中的标准内置标记器相比，使用HuggingFace 的新 Tokenizer 库[41]的性能有所提高。Steven 报告说，其速度提高了 9 倍，并且实现过程花费了 10.6 秒来标记 100 万个句子。
\\
**5.2 Can language models really comprehend?**
\\
The Gradient 最近在Gary Marcus 的这篇文章[42]中发表，他讨论了他认为是 GPT-2 等语言模型背后的基本缺陷的内容。Gary Marcus 的主要观点是，经过训练能够预测下一个单词的模型不一定是可以理解或推理的模型，即“预测是理解的组成部分，而不是整体。” 他还讨论了在语言环境中先天性的重要性，并指出当前的语言模型没有考虑到这一点。
\\
**5.3 Curriculum for Reinforcement Learning**
\\
设计基于课程的方法可以帮助 RL agent 学习吗？Lillian Weng 总结了几种基于课程的方法[43]，以及如何利用它们来培训有效的强化学习代理。Weng 讨论了设计高效的课程学习方法所面临的挑战，该方法通常需要对任务的复杂性进行排序，并向模型提供一系列任务，这些任务会增加培训过程中的难度。
\\
\\
**5.4 Introduction to NumPy**
\\
我总是推荐任何机器学习入门的人来使用 NumPy 进行大量练习。当今，许多用于深度学习和机器学习的高级库都以某种方式使用 NumPy API，因此它是内部了解的非常重要的工具。Anne Bonner 最近发布了这个非常详细的 numpy 教程[44]，介绍了 NumPy 的基础。
\\
# 6、Education 🎓
**6.1 Foundations of machine learning and statistical inference**
\\
来自加州理工学院的 Anima Anandkumar 发布了一门名为“机器学习和统计推论的基础”的课程。该课程侧重于 ML 概念，例如矩阵，张量，优化，概率模型，神经网络等。这是一门很棒的课程，因为它侧重于 ML 的理论方面，这对于理解和改进更高级的方法同样重要。（视频播放列表[45]|课程提纲[46]）
\\
**6.2 Deep Learning Lecture Series**
\\
DeepMind 与 UCL 合作发布了深度学习讲座系列[47]，其中包括 12 个讲座，这些讲座将由 DeepMind 的领先研究科学家进行。主题包括如何使用注意力，记忆力和生成模型等方法训练神经网络。
\\
**6.3 Open Syllabus**
\\
教育是不断发展的社区和整个行业的重要组成部分， 这是种植创新种子的地方。Open Syllabus[48]是一个非营利性组织，它利用众包的力量将高等教育课程映射到一个免费的在线数据库中。它目前包含大约 700 万个教学大纲。
\\
\\
**6.4 Discussing, Sharing, and Learning about ML**
\\
r/ResearchML[49]是用于讨论 ML 论文的新的机器学习子目录。这一主题更侧重于研究并鼓励更深入的讨论。
\\
PracticalAI[50]是一个网站，你可以在其中浏览和发现由社区和专家策划的 ML 主题。
\\
# 7、Notable Mentions ⭐️
How we built the good first issues feature[51]：了解有关 GitHub 如何利用机器学习为开发人员发现简单和个性化问题的更多信息，以便他们可以解决与他们的兴趣相匹配的问题。这鼓励了来自开源贡献者的更快和更多的贡献。
\\
紧跟 Sebastian Ruder 的 NLP News[52]，以获取最新的 NLP 最新新闻。重点包括 NLP 进展的更新，过去十年的回顾，新的 NLP 课程以及其他主题。
\\
一份超赞的TensorFlow 2.0 深度学习 notebook[53]列表，范围从 CycleGAN 到 Transformers 到图像字幕任务。它们由 LBNL 的科学学院深度学习公开发布。
\\
一篇令人印象深刻且易于理解的博客文章，解释了贝叶斯神经网络[54]的基础，入门的绝佳介绍。
\\
An Opinionated Guide to ML Research[55]：John Schulman 就如何更好地选择研究问题以及在实施和解决手头的研究任务方面更具战略性等方面，为即将到来的机器学习研究人员提供了一些建议，还分享了个人发展和持续进步的技巧。
\\
今日限定款分割线，右下角链接可以阅读原文~
# 本文参考资料
----------
[1] **RobBERT:** [https://arxiv.org/abs/2001.06286](https://arxiv.org/abs/2001.06286)
\\
[2] **BERTje:** [https://arxiv.org/abs/1912.09582](https://arxiv.org/abs/1912.09582)
\\
[3] **德语 German BERT:** [https://deepset.ai/german-bert](https://deepset.ai/german-bert)
\\
[4] **葡萄牙语 Portuguese BERT:** [https://github.com/neuralmind-ai/portuguese-bert](https://github.com/neuralmind-ai/portuguese-bert)
\\
[5] **CamemBERT:** [https://arxiv.org/abs/1911.03894](https://arxiv.org/abs/1911.03894)
\\
[6] **FlauBERT:** [https://arxiv.org/abs/1912.05372](https://arxiv.org/abs/1912.05372)
\\
[7] **AlBERTo:** [http://ceur-ws.org/Vol-2481/paper57.pdf](http://ceur-ws.org/Vol-2481/paper57.pdf)
\\
[8] **UmBERTo:** [https://github.com/musixmatchresearch/umberto](https://github.com/musixmatchresearch/umberto)
\\
[9] **BETO:** [https://github.com/dccuchile/beto](https://github.com/dccuchile/beto)
\\
[10] **araBERT:** [https://colab.research.google.com/drive/1KSy89fAkWt6EGfnFQElDjXrBror9lIZh](https://colab.research.google.com/drive/1KSy89fAkWt6EGfnFQElDjXrBror9lIZh)
\\
[11] **Transformer 库:** [https://huggingface.co/models](https://huggingface.co/models)
\\
[12] **2.4.1:** [https://github.com/huggingface/transformers/releases](https://github.com/huggingface/transformers/releases)
\\
[13] **论文:** [https://arxiv.org/abs/2001.06296](https://arxiv.org/abs/2001.06296)
\\
[14] **hallucination :** [https://arxiv.org/abs/1910.08684](https://arxiv.org/abs/1910.08684)
\\
[15] **LaserTagger:** [https://ai.googleblog.com/2020/01/encode-tag-and-realize-controllable-and.html](https://ai.googleblog.com/2020/01/encode-tag-and-realize-controllable-and.html)
\\
[16] **报告:** [https://arxiv.org/abs/2001.07092](https://arxiv.org/abs/2001.07092)
\\
[17] **mBART:** [https://arxiv.org/pdf/2001.08210.pdf](https://arxiv.org/pdf/2001.08210.pdf)
\\
[18] **Meena:** [https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html](https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html)
\\
[19] **关于这项工作的想法:** [https://venturebeat.com/2020/01/31/with-googles-meena-are-ai-assistants-about-to-get-a-lot-smarter/](https://venturebeat.com/2020/01/31/with-googles-meena-are-ai-assistants-about-to-get-a-lot-smarter/)
\\
[20] **Albert Learns to Read:** [https://littlealbert.now.sh/#/](https://littlealbert.now.sh/#/)
\\
[21] **ALBERT:** [https://ai.googleblog.com/2019/12/albert-lite-bert-for-self-supervised.html](https://ai.googleblog.com/2019/12/albert-lite-bert-for-self-supervised.html)
\\
[22] **博客 Teaching Machines to Read:** [https://www.spark64.com/post/machine-comprehension](https://www.spark64.com/post/machine-comprehension)
\\
[23] **A Self-Taught AI Researcher at Google:** [https://blog.floydhub.com/emils-story-as-a-self-taught-ai-researcher/](https://blog.floydhub.com/emils-story-as-a-self-taught-ai-researcher/)
\\
[24] **Google 数据集搜索:** [https://blog.google/products/search/discovering-millions-datasets-web/](https://blog.google/products/search/discovering-millions-datasets-web/)
\\
[25] **Big Bad NLP 数据库:** [https://quantumstat.com/dataset/dataset.html](https://quantumstat.com/dataset/dataset.html)
\\
[26] **PyTorch 库:** [https://github.com/cpnota/autonomous-learning-library](https://github.com/cpnota/autonomous-learning-library)
\\
[27] **Captum:** [https://captum.ai/](https://captum.ai/)
\\
[28] **这套教程:** [https://www.kaggle.com/learn/machine-learning-explainability](https://www.kaggle.com/learn/machine-learning-explainability)
\\
[29] **Flax:** [https://github.com/google-research/flax/tree/prerelease](https://github.com/google-research/flax/tree/prerelease)
\\
[30] **JAX:** [https://github.com/google/jax](https://github.com/google/jax)
\\
[31] **Thinc:** [https://thinc.ai/](https://thinc.ai/)
\\
[32] **Flyte:** [https://eng.lyft.com/introducing-flyte-cloud-native-machine-learning-and-data-processing-platform-fb2bb3046a59](https://eng.lyft.com/introducing-flyte-cloud-native-machine-learning-and-data-processing-platform-fb2bb3046a59)
\\
[33] **DeepPavlov:** [https://github.com/deepmipt/DeepPavlov](https://github.com/deepmipt/DeepPavlov)
\\
[34] **Google Colab:** [https://colab.research.google.com/github/deepmipt/dp_notebooks/blob/master/DP_tf.ipynb](https://colab.research.google.com/github/deepmipt/dp_notebooks/blob/master/DP_tf.ipynb)
\\
[35] **Blog:** [https://medium.com/tensorflow/deeppavlov-an-open-source-library-for-end-to-end-dialog-systems-and-chatbots-31cf26849e37](https://medium.com/tensorflow/deeppavlov-an-open-source-library-for-end-to-end-dialog-systems-and-chatbots-31cf26849e37)
\\
[36] **Demo:** [https://demo.deeppavlov.ai/#/en/textqa](https://demo.deeppavlov.ai/#/en/textqa)
\\
[37] **此处:** [https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html](https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html)
\\
[38] **这个报告:** [https://fortune.com/longform/ai-artificial-intelligence-big-tech-microsoft-alphabet-openai/](https://fortune.com/longform/ai-artificial-intelligence-big-tech-microsoft-alphabet-openai/)
\\
[39] **这份非常详尽的报告:** [https://www.turing.ac.uk/sites/default/files/2019-06/understanding_artificial_intelligence_ethics_and_safety.pdf](https://www.turing.ac.uk/sites/default/files/2019-06/understanding_artificial_intelligence_ethics_and_safety.pdf)
\\
[40] **这篇文章:** [https://towardsdatascience.com/a-small-timing-experiment-on-the-new-tokenizers-library-a-write-up-7caab6f80ea6](https://towardsdatascience.com/a-small-timing-experiment-on-the-new-tokenizers-library-a-write-up-7caab6f80ea6)
\\
[41] **HuggingFace 的新 Tokenizer 库:** [https://github.com/huggingface/tokenizers](https://github.com/huggingface/tokenizers)
\\
[42] **Gary Marcus 的这篇文章:** [https://thegradient.pub/gpt2-and-the-nature-of-intelligence/](https://thegradient.pub/gpt2-and-the-nature-of-intelligence/)
\\
[43] **几种基于课程的方法:** [https://lilianweng.github.io/lil-log/2020/01/29/curriculum-for-reinforcement-learning.html](https://lilianweng.github.io/lil-log/2020/01/29/curriculum-for-reinforcement-learning.html)
\\
[44] **非常详细的 numpy 教程:** [https://numpy.org/devdocs/user/absolute_beginners.html](https://numpy.org/devdocs/user/absolute_beginners.html)
\\
[45] **视频播放列表:** [https://www.youtube.com/playlist?list=PLVNifWxslHCDlbyitaLLYBOAEPbmF1AHg](https://www.youtube.com/playlist?list=PLVNifWxslHCDlbyitaLLYBOAEPbmF1AHg)
\\
[46] **课程提纲:** [http://tensorlab.cms.caltech.edu/users/anima/cms165-2020.html](http://tensorlab.cms.caltech.edu/users/anima/cms165-2020.html)
\\
[47] **深度学习讲座系列:** [https://www.eventbrite.co.uk/o/ucl-x-deepmind-deep-learning-lecture-series-general-29078980901](https://www.eventbrite.co.uk/o/ucl-x-deepmind-deep-learning-lecture-series-general-29078980901)
\\
[48] **Open Syllabus:** [https://opensyllabus.org/](https://opensyllabus.org/)
\\
[49] **r/ResearchML:** [https://www.reddit.com/r/ResearchML/](https://www.reddit.com/r/ResearchML/)
\\
[50] **PracticalAI:** [https://practicalai.me/explore/content/](https://practicalai.me/explore/content/)
\\
[51] **How we built the good first issues feature:** [https://github.blog/2020-01-22-how-we-built-good-first-issues/](https://github.blog/2020-01-22-how-we-built-good-first-issues/)
\\
[52] **NLP News:** [http://newsletter.ruder.io/issues/nlp-progress-restrospectives-and-look-ahead-new-nlp-courses-independent-research-initiatives-interviews-lots-of-resources-217744](http://newsletter.ruder.io/issues/nlp-progress-restrospectives-and-look-ahead-new-nlp-courses-independent-research-initiatives-interviews-lots-of-resources-217744)
\\
[53] **TensorFlow 2.0 深度学习 notebook:** [https://github.com/NERSC/dl4sci-tf-tutorials](https://github.com/NERSC/dl4sci-tf-tutorials)
\\
[54] **贝叶斯神经网络:** [https://engineering.papercup.com/posts/bayesian-neural-nets/](https://engineering.papercup.com/posts/bayesian-neural-nets/)
\\
[55] **An Opinionated Guide to ML Research:** [http://joschu.net/blog/opinionated-guide-ml-research.html](http://joschu.net/blog/opinionated-guide-ml-research.html)
---
## File: posts/_posts/2020-02-16-NLP_Newsletter[PT-BR]_PyTorch3D,_DeepSpeed,_Turing-NLG.md
---
layout: post
title: "NLP Newsletter #4 [PT-BR]: PyTorch3D, DeepSpeed, Turing-NLG, Question Answering Benchmarks, Hydra, Sparse Neural Networks,…"
author: VictorGarritano
modified:
comments: true
excerpt: "Essa edição cobre tópicos como modelos de linguagem maiores, avanços na pesquisa de Deep Learning em 3D, benchmarks para question answering multi-idiomas, auditoria de sistemas de IA e muito mais"
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_4.png
---
# Publicações 📙
***Turing-NLG: A 17-billion-parameter language model by Microsoft***
\\
O Turing Natural Language Generation (T-NLG) é um modelo de linguagem com 17 bilhões de parâmetros [proposto](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/) pelo grupo de pesquisa em Inteligência Artificial da Microsoft. O T-NLG é composto por 78 camadas, baseado na arquitetura dos Transformers, que superou o estado da arte anterior (atribuido ao [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) da Nvidia) considerando a Perplexidade na base de dados WikiText-103. O modelo foi testado em diversas tarefas, como *question answering* e sumarização abstrata, onde foram observados comportamentos interessantes e desejáveis para modelos dessa categoria, como *"zero shot" question answering* (onde o modelo responde a um pergunta sem estar ciente do contexto explicitamente), e baixa necessidade de bases previamente anotadas, para as tarefas citadas anteriormente. O modelo pode ser treinado graças à biblioteca DeepSpeed, utilizando o esquema de otimização ZeRO (que também aparece nessa edição da *Newsletter*).
\\
*Modelos de Linguagem e suas quantidades de parâmetros —* [*fonte*](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/)
\\
***Neural based Dependency Parsing***
\\
A pesquisadora Miryam de Lhoneux liberou recentemente sua tese de Doutorado, entitulada “[Linguistically Informed Neural Dependency Parsing for Typologically Diverse Languages](http://uu.diva-portal.org/smash/record.jsf?pid=diva2%3A1357373&dswid=7905)”. O trabalho desenvolvido propõe a utilização de abordagens baseadas em redes neurais para a tarefa de análise de dependências ([dependency parsing](http://nlpprogress.com/english/dependency_parsing.html)) em idiomas com diversas tipologias (línguas que apresentam padrões funcionais e estruturais diferentes). O trabalho apresentado pela autora indica que a incorporação de RNNs e camadas recursivas nos analisadores pode ser benéfica para a tarefa, uma vez que essas arquiteturas podem indicar o conhecimento linguístico necessário à análise. Outras extensões incluem a utilização de analisadores poliglotas e estratégias de compartilhamento de parâmetros para a análise de linguagens correlacionadas ou descorrelacionadas.
\\
***End-to-end Cloud-based Information Extraction with BERT***
\\
Um time de pesquisadores publicou um [artigo](https://arxiv.org/abs/2002.01861) descrevendo como modelos de Transformers, como o BERT, podem auxiliar sistemas de extração de informação de ponta a ponta em documentos de domínios específicos, como documentação regulatória e de concessão de propriedades. Esse tipo de trabalho, além de otimizar operações de negócios, demonstra a eficiência e aplicabilidade de modelos baseados no BERT em cenários onde bases de dados anotadas são extremamente limitadas. Uma plataforma na nuvem é apresentada e discutida, assim como os detalhes de sua implementação (ver figura abaixo).
\\
[*fonte*](https://arxiv.org/abs/2002.01861)
\\
***Question Answering Benchmark***
\\
Em [Wolfson et al. (2020)](https://arxiv.org/abs/2001.11770v1), apresenta-se um *benchmark* para a tarefa de *question understanding* e um método para decomposição de questões, uma etapa necessária para a determinação de uma resposta apropriada. Os autores recorreram à um serviço de *crowdsourcing* para a criação da base anotada de decomposição de questões. Com objetivo de demonstar a viabilidade e aplicabilidade do método proposto, os autores mostraram que é possível melhorar o desempenho de modelos utilizando essa técnica sobre a base de dados HotPotQA.
\\
*“Questões de diferentes fontes de informações exibem uma estrutura composicional semelhante. Questões em linguagem natural (parte de cima) são decompostas seguindo a metodologia QDMR (meio) e deterministicamente mapeadas para uma linguagem pseudo-formal (parte de baixo).” —* [*fonte*](https://arxiv.org/pdf/2001.11770v1.pdf)
\\
***Radioactive data: tracing through training***
\\
Membros da equipe de pesquisa em IA do Facebook publicaram recentemente um [trabalho interessante](https://ai.facebook.com/blog/using-radioactive-data-to-detect-if-a-data-set-was-used-for-training/) que propõe a marcação de imagens (referenciadas como *radioactivate data*) de tal maneira que seja possível verificar se uma determinada base de dados foi utilizada no treinamento de um modelo de Aprendizado de Máquina. Os autores concluíram que é possível utilizar uma marcação mais robusta, que move as *features* para uma determinada direção, e que pode ser empregada para auxiliar a detecção de dados "radioativos", mesmo quando apenas **1%** destes estão presentes na base de treinamento.
Essa é uma tarefa bem desafiadora, uma vez que qualquer modificação nos dados pode potencialmente prejudicar o desempenho do modelo. De acordo com os autores, o trabalho proposto pode "*ajudar pesquisadores e engenheiros a monitorar quais bases de dados foram utilizadas no treinamento de um modelo, com o objetivo de compreender melhor como bases de dados de diferentes naturezas influenciam o desempenho de diversas redes neurais*". Parece uma tarefa crucial para aplicações *mission-critical*. Confira o artigo completo [aqui](https://arxiv.org/pdf/2002.00937.pdf).
\\
***REALM: Retrieval-Augmented Language Model Pre-Training***
\\
O [REALM](https://kentonl.com/pub/gltpc.2020.pdf) é um método de recuperação em larga escala baseado em redes neurais, que faz uso de bases de conhecimento textual para pré-treinar um modelo de linguagem de maneira não-supervisionada. Essencialmente, o objetivo da abordagem é capturar o conhecimento, de uma maneira mais interpretável, expondo o modelo à conhecimentos gerais utilizados durante o processo de treinamento e inferência através do *backpropagation*. As bases onde o método foi testado e avaliado incluem *benchmarks* de *open-domain question answering*. Além do aumento observado na acurácia do modelo, outros benefícios incluem modularidade e interpretabilidade dos componentes.
\\
[*fonte*](https://kentonl.com/pub/gltpc.2020.pdf)
# Criatividade e Sociedade 🎨
***Apresentações remotas de artigos e pôsteres em conferências científicas***
\\
Durante a semana passada, uma [petição](https://www.change.org/p/organizers-of-data-science-and-machine-learning-conferences-neurips-icml-aistats-iclr-uai-allow-remote-paper-poster-presentations-at-conferences) circulou na internet, reivindicando a permissão para apresentações remotas de artigos e pôsteres em conferências científicas, como as relacionadas à Aprendizado de Máquina. Para saber mais, acesse [change.org](https://www.change.org/p/organizers-of-data-science-and-machine-learning-conferences-neurips-icml-aistats-iclr-uai-allow-remote-paper-poster-presentations-at-conferences). Parece que Yoshua Bengio, um dos pioneiros do *Deep Learning*, está convocando as pessoas à assinar a petição. Ele deixou isso bem claro em seu novo [blog](https://yoshuabengio.org/2020/02/10/fusce-risus/).
\\
***Abstração e Desafios de Raciocínio***
\\
François Chollet postou recentemente uma [competição no Kaggle](https://www.kaggle.com/c/abstraction-and-reasoning-challenge/overview) onde ele disponibilizou o *Abstraction and Reasoning Corpus (ARC)*, uma base de dados que tem como objetivo encorajar os usuários a desenvolver sistemas de IA para resolver tarefas às quais nunca foram expostos. A esperança é que essa competição seja o pontapé inicial para a construção de modelos mais robustos de IA, capazes de resolver novos problemas por conta própria de maneira mais eficiente e rápida, ajudando na resolução de aplicações mais desafiadoras do mundo real como a melhoria de carros autônomos que operam em ambientes diversos e extremos.
\\
***Publicações de Aprendizado de Máquina e Processamento de Linguagem Natural em 2019***
\\
Marek Rei liberou sua [análise anual](https://www.marekrei.com/blog/ml-and-nlp-publications-in-2019/) com estatísticas das publicações sobre ML e NLP em 2019. As conferências consideradas nas análises foram ACL, EMNLP, NAACL, EACL, COLING, TACL, CL, CoNLL, NeurIPS, ICML, ICLR, e AAAI.
\\
***Growing Neural Cellular Automata***
\\
Morfogênese é um processo de auto-organização pelo qual alguns animais, como as salamandras, podem regenerar partes de seus corpos que sofreram danos. O processo é robusto a perturbações e adaptativo na natureza. Inspirado nesse esse fenômeno biológico e com a necessidade de uma melhor compreensão desse mecanismo, pesquisadores publicaram um [trabalho](https://distill.pub/2020/growing-ca/) entitulado “*Growing Neural Cellular Automata*”, que adota um modelo diferenciável para o processo de morfogênese buscando replicar os comportamentos e propriedades de sistemas de auto-reparação.
Espera-se que o processo seja capaz de criar máquinas "auto-reparáveis" que possuam a mesma robustez e maleabilidade dos organismos biológicos. Além disso, o método pode possibilitar um melhor entendimento do processo de regeneração em si. Áreas que podem se beneficiar com essa pesquisa incluem a medicina regenerativa e a modelagem de sistemas sociais e biológicos.
\\
[*fonte*](https://distill.pub/2020/growing-ca/)
\\
***Visualização do mecanismo de Atenção do Transformer***
\\
Hendrik Strobelt compartilhou esse [repositório bem interessante](https://github.com/SIDN-IAP/attnvis) que mostra como construir rapidamente uma visualização simples e interativa da Atenção do Transformer através de uma aplicação web utilizando as bibliotecas Hugging Face e d3.js.
\\
[*fonte*](https://github.com/SIDN-IAP/attnvis)
\\
***SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks***
\\
O SketchTransfer propõe uma nova tarefa que tem por objetivo testar a habilidade de redes neurais profundas em manter a capacidade invariância frente a presença/ausência de detalhes. Um debate de longa data existe acerca da inabilidade de redes profundas em generalizar variações que não foram vistas durante o treinamento, algo que os humanos conseguem fazer com relativa facilidade em situações como, por exemplo, a falta de alguns detalhes visuais quando assistimos desenhos. O trabalho discute e disponibiliza uma base de dados esboços não-anotados e imagens reais anotadas, permitindo que os pesquisadores possam estudar o problema de "*detail-invariance*" de maneira bastante cuidadosa.
\\
[*fonte*](https://arxiv.org/pdf/1912.11570.pdf)
# Bibliotecas e Bases de Dados ⚙️
***DeepSpeed + ZeRO***
\\
A Microsoft liberou um pacote para otimização chamado DeepSpeed, compatível com o PyTorch, que possibilita o treinamento de modelos com 100 bilhões de parâmetros. A biblioteca dá destaque a 4 importantes aspectos do processo de treinamento: *operação em escala*, *velocidade*, *custo*, e *usabilidade*. A DeepSeed foi [liberada](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/) junto com o ZeRO, uma tecnologia que otimiza a utilização da memória, e que possibilita o emprego do Deep Learning em larga escala de maneira distribuída com as atuais tecnologias de GPU, além de melhorar o *throughput* em 3-5 vezes em relação à melhor solução atual. A tecnologia possibilita o treinamento de modelos de tamanho arbitrário que podem ocupar a memória total disponível, distribuída pelos diversos dispositivos na infra-estrutura.
\\
[*fonte*](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/)
\\
***Deep Learning em Superfícies 3D de Maneira Rápida e Eficiente***
\\
A [PyTorch3D](https://ai.facebook.com/blog/-introducing-pytorch3d-an-open-source-library-for-3d-deep-learning/) é uma biblioteca em código aberto para a pesquisa de Deep Learning aplicado à superfícies 3D. Esse pacote, baseado no PyTorch, busca auxiliar no suporte e entendimento de dados em 3D aplicados à redes neurais. A biblioteca consiste de implementações eficientes e otimizadas de operadores e funções de custo 3D comumente utilizadas. Um renderizador diferenciável modular também está disponível, que pode ser útil durante a pesquisa e exploração de entradas 3D com padrões complexos e na geração de predições de alta qualidade.
\\
[*fonte*](https://ai.facebook.com/blog/-introducing-pytorch3d-an-open-source-library-for-3d-deep-learning/)
\\
***Gerenciamento de Configurações para projetos de ML***
\\
Hydra é uma ferramenta de configuração escrita em Python, que auxilia no gerenciamento de projetos complexos de ML de maneira mais eficiente. O propósito é dar suporte a pesquisadores que utilizam o PyTorch, oferencedo a possibilidade de reutilização de configurações de projetos de maneira funcional. O benefício principal oferecido é a possibilidade do programador *compor configurações como compõe-se código*, o que permite a rápida alteração de arquivos de configuração. A Hydra pode ainda gerenciar automaticamente o diretório de trabalho que armazena as saídas do seu projeto de ML, o que é bem útil quando precisamos salvar e acessar diversos resultados provenientes de múltiplos *jobs*. Para saber, mais visite o [site](https://hydra.cc/).
\\
***Uma biblioteca para Inferência Causal com Redes Bayesianas***
\\
A [CausalNex](https://causalnex.readthedocs.io/en/latest/01_introduction/01_introduction.html) é uma ferramenta que busca combinar o aprendizado de máquina e o raciocínio causal, possibilitando a descoberta de relacionamentos estruturais na base de dados. Os autores prepararam um tutorial introdutório que mostra porquê e como inferir causalidade com as Redes Bayesianas utilizando a biblioteca proposta.
\\
[*fonte*](https://causalnex.readthedocs.io/en/latest/01_introduction/01_introduction.html)
\\
***Google Colab Pro agora está disponível***
\\
O Google Colab comecou a oferecer uma versão Pro, que disponibiliza vantagens como o acesso exclusivo à GPUs e TPUs, tempos de execução mais longos e mais memória.
\\
***TyDi QA: A Multilingual Question Answering Benchmark***
\\
O grupo de IA da Google introduziu recentemente a [TyDi QA](https://ai.googleblog.com/2020/02/tydi-qa-multilingual-question-answering.html), uma base de dados multi-idiomas que busca encorajar pesquisadores a abordar a tarefa de *question answering* em línguas mais tipologicamente diversas, ou seja, que apresentam padrões estrturais não-convencionais. A liberação da base visa motivar a construção de modelos mais robustos a idiomas tipologicamente distantes, como o Árabe, Bengali, Coreano, Russo, Telugo e Tailândes, podendo generalizar para outros dialetos.
\\
[*fonte*](https://ai.googleblog.com/2020/02/tydi-qa-multilingual-question-answering.html)
\\
***Question Answering para Node.js***
\\
A empresa Hugging Face liberou uma [biblioteca para *question answering*](https://github.com/huggingface/node-question-answering) baseada no DistilBERT, dando continuidade a sua missão de tornar a área de Processamento de Linguagem Natural mais acessível. O modelo apresentado pode rodar num ambiente de produção utilizando Node.js com apenas 3 linhas de código, se benficiando da implementação eficiente oferecida pela Tokenizers, também desenvolvida pelo Hugging Face, e a versão em Javascript do TensorFlow (TensorFlow.js).
# Ética em IA 🚨
***Identificando viés subjetivo em texto***
\\
Num [podcast](https://podcasts.apple.com/us/podcast/will-ai-help-identify-bias-or-perpetuate-it-with-diyi-yang/id1435564422?i=1000464141922) que contou com a participação de Diyi Yang, um pesquisador em ciência social computacional, foi discutido como sistemas de IA podem auxiliar na identificação de viés subjetivo em informações textuais. Essa é uma área de pesquisa importante envolvendo Inteligência Artificial e NLP, especialmente quando discutimos sobre o consumo de textos, como títulos e chamadas de notícias, e a facilidade que esses meios possuem para influenciar leitores com opiniões subjetivas.
Do ponto de vista da aplicação, se torna de vital importância a tarefa de identificação desse viés de maneira automática, assim como a conscientização dos leitores, para que esses se tornem mais atentos e criteriosos em relação ao conteúdo que estão consumindo.
\\
***Artificial Intelligence, Values and Alignment***
\\
A disseminação de sistemas de IA e a forma com que esses sistemas se alinham com os valores humanos é uma área de pesquisa envolvendo ética na Inteligência Artificial em crescente atividade. A DeepMind publicou recentemente um [artigo](https://deepmind.com/research/publications/Artificial-Intelligence-Values-and-Alignment) que busca investigar de maneira mais profunda as questões filosóficas envolvidas no alinhamento da IA com os valores humanos. O trabalho discute duas frentes: a técnica (*como codificar valores que permitem que agentes de IA produzam resultados confiáveis*), e a normativa (*quais princípios deveriam ser codificados pelos modelos*), e como eles se relacionam e podem ser garantidos. O artigo encoraja uma abordagem baseada em princípios para o alinhamento de valores pelos sistemas de Inteligência, de modo a preservar um tratamento igualitário e justo frente às diferenças de convicções e opiniões.
\\
***Auditoria em Sistemas de IA***
\\
A VentureBeat divulgou que pesquisadores da Google, numa colaboração com outros grupos, criaram um *framework* chamado SMACTR, que permite a auditoria de sistemas de IA. A motivação para esse trabalho envolve a ausência de "prestação de contas" dos sistemas atuais, que são colocados à disposição do público geral. A reportagem completa pode ser acessada [aqui](https://venturebeat.com/2020/01/30/google-researchers-release-audit-framework-to-close-ai-accountability-gap/), assim como o [trabalho completo](https://dl.acm.org/doi/abs/10.1145/3351095.3372873).
# Artigos e Postagens ✍️
***Destilação de modelos em sistemas de NLP***
\\
Num [novo episódio](https://soundcloud.com/nlp-highlights/104-model-distillation-with-victor-sanh-and-thomas-wolf) do podcast *NLP Highlights*, Thomas Wolf and Victor Sanh discutiram sobre a destilação de modelos e como a técnica pode ser utilizada como uma alternativa factível para a compressão de grandes arquiteturas, como o BERT, para aplicações escaláveis de NLP em cenários reais. A metodologia é discutida no trabalho publicado pelos convidados, entitulado [DistilBERT](https://arxiv.org/abs/1910.01108), onde são construídos modelos menores (baseados na mesma arquitetura do modelo original) que tentam simluar o comportamento do modelo com maior número de parâmetros, de acordo com suas saídas. Essencialmente, o menor modelo (*student*) tenta modelar a distribuição de probabilidade do modelo maior (*teacher*) baseado na distribuição empírica gerado por suas saídas.
\\
***BERT, ELMo, & GPT-2: How contextual are contextualized word representations?***
\\
O sucesso de métodos contextualizados como o BERT para resolução de uma ampla gama de tarefas complexas de NLP é um assunto que está em voga no momento. Nesse [post](https://kawine.github.io/blog/nlp/2020/02/03/contextual.html), Kawin Ethayarajh tenta responder a questão que diz respeito à quão contextuais os modelos como BERT, o ELMo e o GPT-2 e seus respectivos *word embedddings* contextualizados são. As características exploradas incluem métricas de contextualidade, especificidade de contexto, além de comparações entre representações vetoriais de palavras "estáticas" e suas versões contextualizadas.
\\
[*fonte*](https://kawine.github.io/blog/nlp/2020/02/03/contextual.html)
\\
***Esparsidade em Redes Neurais***
\\
François Lagunas, pesquisador na área de ML, escreveu esse excelente [post](https://medium.com/huggingface/is-the-future-of-neural-networks-sparse-an-introduction-1-n-d03923ecbd70) compartilhando seu otimismo em relação à utilização de tensores esparsos em modelos de redes neurais. A expectativa é empregar alguma forma de esparsidade visando a redução do tamanho dos modelos atuais, que de certa forma estão se tornando impraticáveis, dadas suas colossais quantidades de parâmetros. Os Transformers, por exemplo, com seus bilhões de parâmetros, poderiam se beneficar com o emprego dessa técnica.
Entretanto, os detalhes de implementação para viabilizar a utilização eficiente da esparsidade em GPU ainda não estão claros... Felizmente, a comunidade de Aprendizado de Máquina já está trabalhando nisso!
\\
***Treinando Seu Próprio Modelo de Linguagem***
\\
Se você está interessado em aprender como treinar um modelo de linguagem do zero, confira esse excelente [tutorial](https://huggingface.co/blog/how-to-train) da Hugging Face que utiliza as suas incríveis bibliotecas Tokenizers e Transformers no treinamento do modelo.
\\
***Tokenizers: How machines read***
\\
Cathal Horan publicou um [blog post](https://blog.floydhub.com/tokenization-nlp/) impresssionante e bem detalhado sobre como e quais tipos de *tokenizers* vêm sendo utilizados nos mais recentes modelos de NLP, auxiliando modelos de Inteligência a aprender por meio de informações textuais. O post também discute e motiva porquê a tarefa de tokenização é uma importante e desafiadora área de pesquisa ativa. O artigo apresenta ainda como treinar o seu próprio *tokenizer* utilizando métodos como o SentencePiece e o WordPiece.
\\
[*fonte*](https://blog.floydhub.com/tokenization-nlp/%27)
# Educação 🎓
***Machine Learning na VU Amsterdam***
\\
Agora você pode acompanahar o curso [2020 MLVU machine learning](https://mlvu.github.io/) pela internet, onde estão inclusas a lista completa de slides, [videos](https://www.youtube.com/watch?v=excCZSTJEPs&feature=youtu.be) e o plano de estudos. O curso oferece uma introdução à ML, além de cobrir tópicos mais avançados de Deep Learning, como *Variational AutoEncoders* (VAEs) e Redes Neurais Adversariais (GANs).
\\
[*fonte*](https://mlvu.github.io/)
\\
***Materiais de Matemática para ML***
\\
Suzana Ilić e a organização Machine Learning Tokyo (MLT) vêm realizando um excelente trabalho em prol da democratização do conhecimento em ML. Confira esse [repositório](https://github.com/Machine-Learning-Tokyo/Math_resources) que apresenta uma coleção de fontes e materiais sobre os fundamentos matemáticos utilizados em Aprendizado de Máquina.
\\
***Introduction to Deep Learning***
\\
Acompanhe o curso “[Introduction to Deep Learning](http://introtodeeplearning.com/)” do MIT nesse site. Novas aulas serão postadas toda semanas e todos os materiais, como slides, vídeos e códigos utilizados, serão publicados.
\\
***Deep Learning com PyTorch***
\\
Alfredo Canziani publicou os slides e notebooks utilizados no minicurso de Deep Learning com Pytorch. O repositório contém ainda um [site](https://atcold.github.io/pytorch-Deep-Learning/) que incluem notas sobre os conceitos apresentados no curso.
\\
***Missing Semester of Your CS***
\\
O “[Missing Semester of Your CS](https://missing.csail.mit.edu/)” é um excelente curso online composto por recursos que podem ser potencialmente úteis para cientistas de dados que não possuem background na área de desenvolvimenteo. Estão inclusos materiais sobre *shell scripting* e versionamento. O curso foi disponibilizado por alunos do MIT.
\\
[*fonte*](https://missing.csail.mit.edu/2020/shell-tools/)
\\
***Advanced Deep Learning***
\\
A CMU disponibilizou recentemente os slides e plano de estudos para o curso “[Advanced Deep Learning](https://andrejristeski.github.io/10707-S20/syllabus.html)”, que cobre tópicos como modelos autoregressivos, modelos generativos, aprendizado autosupervisionado, entre outros. O público-alvo são alunos de mestrado e doutorado com sólidos conhecimentos de ML.
# Menções honrosas ⭐️
\\
Você pode encontrar a última Newsletter [aqui](https://medium.com/dair-ai/nlp-newsletter-flax-thinc-language-specific-bert-models-meena-flyte-lasertagger-4f7da04a9060). Essa edição cobriu tópicos como melhorias em agentes conversacionais, divulgação de modelos BERT para idiomas específicos (entre eles o **Português**!!!), bases de dados publicamente disponívies, introdução de novas bibliotecas para Deep Learning, e muito mais.
Em Xu et al. (2020), foi proposto um [método](https://arxiv.org/abs/2002.02925]) para progressivamente substituir e comprimir modelos BERT através da separação de seus componentes originais. Através dessa substituição progressiva, aliado ao processo de treinamento, é possível combinar os componentes originais e suas versões compactadas. A metodologia apresentada supera outras abordagens de *knowledge distillation* no benchmark GLUE.
Um outro curso interessante é o “[Introduction to Machine Learning](https://compstat-lmu.github.io/lecture_i2ml/index.html)”, que cobre o básico de ML, regressão supervisionada, *random forests*, ajuste de parâmetros, dentre outros conceitos fundamentais.
A versão para 🇬🇷 grego do BERT ([GreekBERT](https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1)) está disponível para uso através da biblioteca Transformers, da Hugging Face.
Jeremy Howard publicou um [artigo](https://arxiv.org/abs/2002.04688) descrevendo a biblioteca de Deep Learning fast.ai, que é amplamente utilizada para pesquisa e ensino em seus cursos de Deep Learning. Uma leitura bastante recomendada para desenvolvedores de software que trabalham construindo e melhorando pacotes de Aprendizado de Máquina e Deep Learning.
A Deeplearning.ai completou o lançamento dos seus 4 cursos da série [TensorFlow: Data and Deployment Specialization](https://www.coursera.org/specializations/tensorflow-data-and-deployment). A especialização visa ensinar desenvolverdores a como realizar o *deploy* de modelos de maneira efetiva e eficiente nos mais diferentes cenários, além de utilizar dados de maneiras eficazes durante o treinamento de modelos.
Sebastian Raschka publicou recentemente um [artigo](https://arxiv.org/abs/2002.04803) entitulado “Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence”. O trabalho apresenta uma revisão bastante acessível às mais variadas ferramentas de ML. É um artigo excelente para compreender as vantagens de algumas bibliotecas e conceitos utilizados em Aprendizado de Máquina. Além disso, levanta-se a discussão sobre o futuro de bibliotecas de ML baseadas em Python.
---
## File: posts/_posts/2020-02-16-NLP_Newsletter_PyTorch3D,_DeepSpeed,_Turing-NLG.md
---
layout: post
title: "NLP Newsletter #4 [EN]: PyTorch3D, DeepSpeed, Turing-NLG, Question Answering Benchmarks, Hydra, Sparse Neural Networks,…"
author: billy_rick
modified:
comments: true
excerpt: "This issue covers topics such as bigger language models, improving 3D deep learning research, multilingual question answering benchmark, auditing AI systems, and much more."
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_4.png
---
# Publications 📙
***Turing-NLG: A 17-billion-parameter language model by Microsoft***
\\
Turing Natural Language Generation (T-NLG) is a 17-billion-parameter language model [proposed](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/) by Microsoft AI researchers. Besides being the largest known language model to date (depicted in the figure below), T-NLG is a 78-layers Transformer-based language model that outperforms the previous state-of-the-art results (held by NVIDIA’s [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)) on WikiText-103 perplexity. It was tested on a variety of tasks such as question answering and abstractive summarization while demonstrating desirable benefits such as zero short question capabilities and minimizing supervision, respectively. The model is made possible by a training optimization library called DeepSpeed with ZeRO, which is also featured later in this newsletter.
\\
*Language model sizes represented by parameters —* [*source*](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/)
\\
***Neural based Dependency Parsing***
\\
Miryam de Lhoneux released her Ph.D. thesis titled “[Linguistically Informed Neural Dependency Parsing for Typologically Diverse Languages](http://uu.diva-portal.org/smash/record.jsf?pid=diva2%3A1357373&dswid=7905)”. This work is about using neural approaches for [dependency parsing](http://nlpprogress.com/english/dependency_parsing.html) in typologically diverse languages (i.e. languages that construct and express meaning in structurally different ways). This paper reports that RNNs and recursive layers could be beneficial for incorporating into parsers as they help to inform models with important linguistic knowledge needed for parsing. Other ideas include the use of polyglot parsing and parameter sharing strategies for parsing in related and unrelated languages.
\\
***End-to-end Cloud-based Information Extraction with BERT***
\\
A team of researchers published a [paper](https://arxiv.org/abs/2002.01861) describing how Transformer models like BERT can help for end-to-end information extraction in domain-specific business documents such as regulatory filings and property lease agreements. Not only can this type of work help to optimize business operations but it also shows the applicability and effectiveness of BERT-based models on regimes with very low annotated data. An application, and its implementation details, that operates on the cloud is also proposed and discussed (see figure below).
\\
[*source*](https://arxiv.org/abs/2002.01861)
\\
***Question Answering Benchmark***
\\
[Wolfson et al. (2020)](https://arxiv.org/abs/2001.11770v1) published a question understanding benchmark and a method for breaking down a question that is necessary for computing an appropriate answer. They leverage crowdsourcing to annotate the required steps needed to break down questions. To show the feasibility and applicability of the approach, they improve on open-domain question answering using the HotPotQA dataset.
\\
\\
*“Questions over different sources share a similar compositional structure. Natural language questions from multiple sources (top) are annotated with the QDMR formalism (middle) and deterministically mapped into a pseudo-formal language (bottom).” —* [*source*](https://arxiv.org/pdf/2001.11770v1.pdf)
\\
***Radioactive data: tracing through training***
\\
Facebook AI researchers recently published [an interesting work](https://ai.facebook.com/blog/using-radioactive-data-to-detect-if-a-data-set-was-used-for-training/) that aims to mark images (referred to as radioactive data) so as to verify if that particular data set was used for training the ML model. They found that it is possible to use a clever marker that moves features towards a direction, which the model uses to help detect the usage of radioactive data even when only 1 percent of the training data is radioactive. This is challenging since any change in the data can potentially degrade the model accuracy. According to the authors, this work can “*help researchers and engineers to keep track of which data set was used to train a model so they can better understand how various data sets affect the performance of different neural networks*”. It seems like an important approach in mission-critical ML applications. Check out the full paper [here](https://arxiv.org/pdf/2002.00937.pdf).
\\
***REALM: Retrieval-Augmented Language Model Pre-Training***
\\
[REALM](https://kentonl.com/pub/gltpc.2020.pdf) is a large-scale neural-based retrieval approach that makes use of a corpus of textual knowledge to pre-train a language model in an unsupervised manner. This approach essentially aims to capture knowledge in a more interpretable way by exposing the model to world knowledge that is used for training and predictions via backpropagation. Tasks approached and evaluated using REALM include open-domain question answering benchmarks. Besides the improvements in the accuracy of the model, other benefits include the modularity and interpretability components.
\\
[*source*](https://kentonl.com/pub/gltpc.2020.pdf)
# Creativity and Society 🎨
***Allowing remote paper & poster presentations at scientific conferences***
\\
The past week there was the circulation of a [petition](https://www.change.org/p/organizers-of-data-science-and-machine-learning-conferences-neurips-icml-aistats-iclr-uai-allow-remote-paper-poster-presentations-at-conferences) to allow for remote paper and poster presentations at scientific conferences like ML related ones. Go read more about it on [change.org](https://www.change.org/p/organizers-of-data-science-and-machine-learning-conferences-neurips-icml-aistats-iclr-uai-allow-remote-paper-poster-presentations-at-conferences). It seems Yoshua Bengio, a pioneer in deep learning, is advocating for people to go and sign the petition. He made this clear in his new [blog](https://yoshuabengio.org/2020/02/10/fusce-risus/).
\\
***Abstraction and Reasoning Challenge***
\\
François Chollet has recently posted a [Kaggle competition](https://www.kaggle.com/c/abstraction-and-reasoning-challenge/overview) where he released the Abstraction and Reasoning Corpus (ARC) that aims to encourage users to create AI systems that can solve reasoning tasks it has never been exposed to. The hope is to begin to build more robust AI systems that are able to better and quickly solve new problems on its own which could help to address the more challenging real-world applications such as improving self-driving cars that operate in extreme and diverse environments.
\\
***ML and NLP Publications in 2019***
\\
Marek Rei releases his [yearly analysis](https://www.marekrei.com/blog/ml-and-nlp-publications-in-2019/) of machine learning and NLP publication statistics from 2019. The conferences included in the analysis are ACL, EMNLP, NAACL, EACL, COLING, TACL, CL, CoNLL, NeurIPS, ICML, ICLR, and AAAI.
\\
***Growing Neural Cellular Automata***
\\
Morphogenesis is a self-organization process by which some creatures such as salamanders can regenerate or repair body damage. The process is robust to perturbations and adaptive in nature. Inspired by this biological phenomenon and a need to understand the process better, researchers published a [paper](https://distill.pub/2020/growing-ca/) titled “Growing Neural Cellular Automata”, which adopts a differentiable model for morphogenesis that aims to replicate behaviors and properties of self-repairing systems. The hope is to be able to build self-repairing machines that possess the same robustness and plasticity as biological life. In addition, it would help to better understand the process of regeneration itself. Applications that can benefit include regenerative medicine and modeling of social and biological systems.
\\
[*source*](https://distill.pub/2020/growing-ca/)
\\
***Visualizing Transformer attention***
\\
Hendrik Strobelt shared this [great repository](https://github.com/SIDN-IAP/attnvis) that shows how to quickly build a simple interactive Transformer attention visualization through a web application by using the Hugging Face library and d3.js.
\\
[*source*](https://github.com/SIDN-IAP/attnvis)
\\
***SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks***
\\
SketchTransfer proposes a new task to test the ability of deep neural networks to support invariance in the presence/absence of details. It has long been debated that deep networks cannot generalize to variations that have not yet been seen during training, something humans can do with relative ease such as dealing with the missing visual details when watching cartoons. The paper discusses and releases a dataset to help researchers carefully study the “detail-invariance” problem by providing unlabelled sketch images and labeled examples of real images.
\\
[*source*](https://arxiv.org/pdf/1912.11570.pdf)
# Tools and Datasets ⚙️
***DeepSpeed + ZeRO***
\\
Microsoft open sources a training optimization library called DeepSpeed, which is compatible with PyTorch and can enable the ability to train a 100-billion-parameter model. The library focuses on four important aspects of training a model: *scale*, *speed*, *cost*, and *usability*. DeepSpeed was [released](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/) together with ZeRO which is a memory optimization technology for enabling large-scale distributed deep learning in current GPU technology while improving throughput three to five times more than the best current system. ZeRO allows the training of models with any arbitrary size that can fit in the aggregated available memory in terms of shared model states.
\\
[*source*](https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/)
\\
***A library for conducting fast and efficient 3D deep learning research***
\\
[PyTorch3D](https://ai.facebook.com/blog/-introducing-pytorch3d-an-open-source-library-for-3d-deep-learning/) is an open-source toolkit for 3D based deep learning research. This PyTorch library aims to help with the support and understanding of 3D data in deep learning systems. The library consists of fast and optimized implementations of frequently used 3D operators and loss functions. It also comes with a modular differentiable renderer which helps to conduct research on complex 3D inputs and supports high-quality 3D predictions.
\\
[*source*](https://ai.facebook.com/blog/-introducing-pytorch3d-an-open-source-library-for-3d-deep-learning/)
\\
***Managing Configuration of ML projects***
\\
Hydra is a Python-based configuration tool for more efficiently managing complex ML projects. It is meant to help PyTorch researchers by offering functional reuse of configurations for ML projects. The main benefit it offers is that it allows the programmer to *compose the configuration like composing code*, which means the configuration file can be easily overridden. Hydra can also help with automatically managing the working directory of your ML project outputs which is useful when needing to save and accessing the results of several experiments for multiple jobs. Learn more about it [here](https://medium.com/pytorch/hydra-a-fresh-look-at-configuration-for-machine-learning-projects-50583186b710).
\\
***A Toolkit for Causal Inferencing with Bayesian Networks***
\\
[CausalNex](https://causalnex.readthedocs.io/en/latest/01_introduction/01_introduction.html) is a toolkit for “causal inference with Bayesian Networks”. The tool aims to combine machine learning and causal reasoning for uncovering structural relationships in data. The authors also prepared an introductory guide on why and how to infer causation with Bayesian networks using the proposed Python library.
\\
[*source*](https://causalnex.readthedocs.io/en/latest/01_introduction/01_introduction.html)
\\
***Google Colab Pro is now available***
\\
Google Colab is now offering a Pro edition, which offers advantages such as exclusive access to faster GPUs and TPUs, longer runtimes, and more memory.
\\
***TyDi QA: A Multilingual Question Answering Benchmark***
\\
Google AI releases [TyDi QA](https://ai.googleblog.com/2020/02/tydi-qa-multilingual-question-answering.html) which is a multilingual dataset that can encourage researchers to perform question answering on more typologically diverse languages that construct and express meaning in different ways. The idea is to motivate researchers to build more robust models on typologically distant languages, such as Arabic, Bengali, Korean, Russian, Telugu, and Thai, so as to generalize to even more languages.
\\
[*source*](https://ai.googleblog.com/2020/02/tydi-qa-multilingual-question-answering.html)
\\
***Question Answering for Node.js***
\\
Hugging Face releases a [question answering library](https://github.com/huggingface/node-question-answering) based on DistilBERT and continues to make NLP more accessible. This model can run in production using Node.js with just 3 lines of code. The model leverages the fast implementation of Tokenizers, also built by Hugging Face, and TensorFlow.js (a popular library for using machine learning models with Javascript).
# Ethics in AI 🚨
***Identifying subjective bias in text***
\\
This [podcast episode](https://podcasts.apple.com/us/podcast/will-ai-help-identify-bias-or-perpetuate-it-with-diyi-yang/id1435564422?i=1000464141922) features Diyi Yang, a researcher in computational social science, who talks about how AI systems can help to identify subjective bias in textual information. This is an important area of research involving AI systems and NLP especially when we discuss the consumption of text media such as news headlines that can be easily framed to bias consumers when in reality they should aim to be more objective. From an application perspective, it becomes critical to automatically identify the subjective bias present in text media so as to help consumers become more aware of the content they are consuming. The episode also discusses how AI can also perpetuate bias.
\\
***Artificial Intelligence, Values and Alignment***
\\
The rise of AI systems and how they align human values is an active area of research that involves ethics in AI systems. DeepMind recently released a [paper](https://deepmind.com/research/publications/Artificial-Intelligence-Values-and-Alignment) that takes a deeper look at the philosophical questions surrounding AI alignment. The report focuses on discussing two parts, technical (i.e., *how to encode values that render reliable results from AI agents*) and normative (*what principles would be right to encode in AI*), and how they relate and can be ensured. The paper pushes for a principle-based approach for AI alignment and to preserve fair treatment despite the difference in beliefs and opinions.
\\
***On Auditing AI Systems***
\\
VentureBeat reports that Google Researchers, in collaboration with other groups, created a framework called SMACTR that allows engineers to audit AI systems. The reason for this work is to address the accountability gap that exists with current AI systems that are put in the wild to be used by consumers. Read the full report [here](https://venturebeat.com/2020/01/30/google-researchers-release-audit-framework-to-close-ai-accountability-gap/) and the full paper [here](https://dl.acm.org/doi/abs/10.1145/3351095.3372873).
# Articles and Blog posts ✍️
***On model distillation for NLP systems***
\\
In a [new episode](https://soundcloud.com/nlp-highlights/104-model-distillation-with-victor-sanh-and-thomas-wolf) of the NLP Highlights podcast, Thomas Wolf and Victor Sanh talk about model distillation and how it can be used as a feasible approach to compress large models like BERT for scalable real-world NLP based applications. This concept is further discussed in their proposed method called [DistilBERT](https://arxiv.org/abs/1910.01108) where they build smaller models (based on the same architecture of a bigger model) to try to mimic the behavior of the bigger model based on the output of that model. In essence, the smaller model (student) tries to fit the probability distribution of the teacher based on its output distribution.
\\
***BERT, ELMo, & GPT-2: How contextual are contextualized word representations?***
\\
Recently there has been a lot of talk on the success of contextualized methods like BERT for approaching a wide variety of complex NLP tasks. In this [post](https://kawine.github.io/blog/nlp/2020/02/03/contextual.html), Kawin Ethayarajh attempts to answer the question of how contextual models like BERT, ELMo and GPT-2 and their contextualized word representation are? Topics include measures of contextuality, context-specificity, and comparisons between static embeddings and contextualized representations.
\\
[*source*](https://kawine.github.io/blog/nlp/2020/02/03/contextual.html)
\\
***Sparsity in Neural Networks***
\\
François Lagunas, an ML researcher, wrote this great [blog post](https://medium.com/huggingface/is-the-future-of-neural-networks-sparse-an-introduction-1-n-d03923ecbd70) discussing his optimism for adopting sparse tensors in neural network models. The hope is to employ some form of sparsity to reduce the size of current models that at some point become unpractical due to their size and speed. This concept may be worth exploring in ML due to the sheer size of current models like Transformers (often relying on billions of parameters). However, the implementation details to support efficient sparsity in neural networks on GPUs are not so clear from a developer tool perspective and that is something the machine learning community is working on already.
\\
***Training your own language model***
\\
If you are interested to learn how to train a language model scratch, check out this impressive and comprehensive [tutorial](https://huggingface.co/blog/how-to-train) by Hugging Face. They obviously leverage their own libraries Transformers and Tokenizers to train the model.
\\
***Tokenizers: How machines read***
\\
Cathal Horan published an impressive and very detailed [blog post](https://blog.floydhub.com/tokenization-nlp/) about how and what type of tokenizers are being used by the most recent NLP models to help machine learning algorithms learn from textual information. He also discusses and motivated why tokenization is an exciting and important active area of research. The article even shows you how to train your own tokenizers using tokenization methods like SentencePiece and WordPiece.
\\
[*source*](https://blog.floydhub.com/tokenization-nlp/%27)
# Education 🎓
***Machine Learning at VU University Amsterdam***
\\
You can now follow the [2020 MLVU machine learning](https://mlvu.github.io/) course online, which includes the full set of slides, [videos](https://www.youtube.com/watch?v=excCZSTJEPs&feature=youtu.be), and syllabus. It is meant to be an introduction to ML but it also has other deep learning related topics such as VAEs and GANs.
\\
[*source*](https://mlvu.github.io/)
\\
***Math resources for ML***
\\
Suzana Ilić and the Machine Learning Tokyo (MLT) have been doing amazing work in terms of democratizing ML education. For example, check out this [repository](https://github.com/Machine-Learning-Tokyo/Math_resources) showcasing a collection of free online resources for learning about the foundations of mathematical concepts used in ML.
\\
***Introduction to Deep Learning***
\\
Keep track of the “[Introduction to Deep Learning](http://introtodeeplearning.com/)” course by MIT on this website. New lectures will be posted every week and all the sides and videos, including coding labs, will be published.
\\
***Deep Learning with PyTorch***
\\
Alfredo Canziani has published the slides and notebooks for the minicourse on Deep Learning with PyTorch. The repository also contains a [companion website](https://atcold.github.io/pytorch-Deep-Learning/) that includes text descriptions of the concepts taught in the course.
\\
***Missing Semester of Your CS***
\\
The “[Missing Semester of Your CS](https://missing.csail.mit.edu/)” is a great online course that consists of material that could potentially be useful for data scientists with non-dev backgrounds. It includes topics such as shell tools and scripting and version control. The course was published by MIT faculty members.
\\
[*source*](https://missing.csail.mit.edu/2020/shell-tools/)
\\
***Advanced Deep Learning***
\\
CMU released the slides and syllabus for the “[Advanced Deep Learning](https://andrejristeski.github.io/10707-S20/syllabus.html)” course which includes topics such as autoregressive models, generative models, and self-supervised/predictive learning, among others. The course is meant for MS or Ph.D. students with an advanced background in ML.
# Noteworthy Mentions ⭐️
You can catch the previous NLP Newsletter here. The [issue](https://medium.com/dair-ai/nlp-newsletter-flax-thinc-language-specific-bert-models-meena-flyte-lasertagger-4f7da04a9060) covers topics such as improving conversational agents, releases of language-specific BERT models, free datasets, releases of deep learning libraries, and much more.
\\
Xu et al. (2020) proposed a [method](https://arxiv.org/abs/2002.02925) for progressively replacing and compressing a BERT model by dividing it into its original components. Through progressive replacement and training, there is also the advantage of combining the original components and compacted versions of the model. The proposed model outperforms other knowledge distillation approaches on the GLUE benchmark.
\\
Here is another interesting course called “[Introduction to Machine Learning](https://compstat-lmu.github.io/lecture_i2ml/index.html)” which covers the ML basics, supervised regression, random forests, parameter tuning, and many more fundamental ML topics.
\\
🇬🇷 Greek BERT ([GreekBERT](https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1)) model is now available for use through the Hugging Face Transformers library.
\\
Jeremy Howard publishes a [paper](https://arxiv.org/abs/2002.04688) describing the fastai deep learning library which is widely used for research and to teach their open courses on deep learning. A recommended read for software developers working on building and improving deep learning and ML libraries.
\\
Deeplearning.ai completes the release of all four courses of the [TensorFlow: Data and Deployment Specialization](https://www.coursera.org/specializations/tensorflow-data-and-deployment). The specialization mainly aims to educate developers on how to efficiently and effectively deploy models in different scenarios and make use of data in interesting and effective ways while training models.
\\
Sebastian Raschka recently published a [paper](https://arxiv.org/abs/2002.04803) titled “Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence”. The paper serves as a comprehensive review of the machine learning tools landscape. It is an excellent report for understanding the various advantages of some libraries and concepts used in ML engineering. In addition, a word on the future of Python-based machine learning libraries is provided.
---
## File: posts/_posts/2020-02-17-NLP简报_ISSUE_4_PyTorch3D,_DeepSpeed,_Turing-NLG.md
---
layout: post
title: "NLP简报 #4 [CH]: PyTorch3D, DeepSpeed, Turing-NLG, Question Answering Benchmarks, Hydra, Sparse Neural Networks,…"
author: billy_rick
modified:
comments: true
excerpt: ""
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_4.png
---
\\
*Thanks to* [*Kaiyuan*](https://blog.csdn.net/Kaiyuan_sjtu) *(WeChat: NewBeeNLP) for this great translation.*
\\
欢迎来到 NLP 时事简报！全文较长，建议收藏。
\\
另外加了目录方便直接索引到自己感兴趣的部分。enjoy
- **1、Publications 📙**
- 1.1 Turing-NLG:
- 1.2 Neural based Dependency Parsing
- 1.3 End-to-end Cloud-based Information Extraction with BERT
- 1.4 Question Answering Benchmark
- 1.5 Radioactive data: tracing through training
- 1.6 REALM: Retrieval-Augmented Language Model Pre-Training
- **2、Creativity and Society 🎨**
- 2.1 允许在科学会议上进行远程论文和海报展示
- 2.2 Abstraction and Reasoning Challenge
- 2.3 ML and NLP Publications in 2019
- 2.4 Growing Neural Cellular Automata
- 2.5 Transformer attention可视化
- 2.6 SketchTransfer
- **3、Tools and Datasets ⚙️**
- 3.1 DeepSpeed + ZeRO
- 3.2 一个用于进行快速有效的3D深度学习研究的库
- 3.3 管理你的ML项目配置
- 3.4 贝叶斯网络因果推理工具包
- 3.5 TyDi问答：多语言问答基准
- 3.6 Question Answering for Node.js
- **4、Ethics in AI 🚨**
- 4.1 识别文本中的subjective bias
- 4.2 Artificial Intelligence, Values and Alignment
- 4.3 关于审核AI系统
- **5、Articles and Blog posts ✍️**
- 5.1 用于NLP系统的模型蒸馏
- 5.2 单词的上下文表示
- 5.3 神经网络中的稀疏性
- 5.4 训练你自己的语言模型
- 5.5 分词器Tokenizer
- **6、Education 🎓**
- 6.1 阿姆斯特丹自由大学机器学习课程
- 6.2 机器学习数学资源
- 6.3 深度学习入门
- 6.4 Pytorch深度学习
- 6.5 Missing Semester of Your CS
- 6.6 深度学习进阶
- **7、Noteworthy Mentions ⭐️**
# 1、Publications 📙
**1.1 Turing-NLG: A 17-billion-parameter language model by Microsoft**
\\
图灵自然语言生成（T-NLG）[1]是由 Microsoft AI 研究人员提出的 170 亿参数语言模型。除了是迄今为止最大的已知语言模型（如下图所示）之外，T-NLG 是基于 78 层 Transformer 的语言模型，其在 WikiText-103 上的困惑度性能优于之前的最新技术成果（由NVIDIA Megatron-LM[2]持有） 。T-NLG 在各种任务（例如问题回答和抽象摘要）上进行了测试，同时分别显示了模型的好处，例如零简短问题功能和最小化监督。此外，该模型得益于 DeepSpeed 库（与 PyTorch 兼容）和 ZeRO 优化器，这两者也会在本期简报中具体介绍。
\\
*语言模型参数大小*
\\
**1.2 Neural based Dependency Parsing**
\\
Miryam de Lhoneux 公布了其博士学位论文“Linguistically Informed Neural Dependency Parsing for Typologically Diverse Languages[3]”。这项工作是关于使用神经学方法以类型多样的语言（即以结构上不同的方式构造和表达含义的语言）进行依赖关系解析[4]。论文指出 RNN 和递归层可能有助于合并到解析器中，因为它们有助于告知具有解析所需的重要语言知识的模型。更多 ideas 包括使用多语言解析和参数共享策略来解析相关和不相关语言。
\\
**1.3 End-to-end Cloud-based Information Extraction with BERT**
\\
研究人员发表了一篇论文"Rapid Adaptation of BERT for Information Extraction on Domain-Specific Business Documents"[5]，描述了 BERT 等 Transformer 模型如何帮助特定领域的业务文档（例如监管文件和物业租赁协议）中的端到端信息提取。这种类型的工作不仅可以帮助优化业务运营，而且还显示了基于 BERT 的模型在带注释数据很少的情况下的适用性和有效性。同时提出并讨论了在云上运行的应用程序及其实现细节（请参见下图）。
\\
\\
**1.4 Question Answering Benchmark**
\\
Wolfson 等发布了一个question understanding benchmark[6]，以及一种用于分解计算适当答案所必需的问题的方法。他们利用众包来注释分解问题所需的必要步骤， 为了展示该方法的可行性和适用性，他们改进了使用 HotPotQA 数据集的开放域问答。
\\
*不同来源的问题具有相似的组成结构。*
\\
**1.5 Radioactive data: tracing through training**
\\
Facebook AI 研究人员最近发表了一项有趣的工作[7]，旨在标记图像（称为**「radioactive data」**），以验证该特定数据集是否用于训练 ML 模型。他们发现，可以使用巧妙的标记将特征移向某个方向，即使只有 1％的训练数据是 radioactiv，模型也可以使用该标记帮助检测‘radioactive data’的使用情况。这极具挑战性，因为数据中的任何更改都可能会降低模型的准确性。作者说，这项工作可以“帮助研究人员和工程师跟踪用于训练模型的数据集，以便他们可以更好地了解各种数据集如何影响不同神经网络的性能”，在关键任务 ML 应用程序中，这似乎是一种重要的方法。如果感兴趣可以查看完整论文[8]了解具体信息。
\\
**1.6 REALM: Retrieval-Augmented Language Model Pre-Training**
\\
REALM [9] 是基于神经的大规模检索方法，该方法利用文本知识的语料库以无监督的方式预训练语言模型。该方法的主要目的是以一种更可解释的方式捕获知识，具体而言是利用世界知识对模型进行训练和预测。使用 REALM 处理和评估的任务包括开放域问答基准， 除了提高模型的准确性外，其他好处还包括模块化和可解释性组件。
\\
# 2、Creativity and Society 🎨
**2.1 允许在科学会议上进行远程论文和海报展示**
\\
过去一周有请愿书散发，以便在与 ML 相关的科学会议上进行远程论文和海报展示，可以在change.org[10]上阅读有关它的更多信息。深度学习的先驱 Yoshua Bengio 似乎在倡导人们去签署请愿书， 并在他的新博客[11]中阐明了这一点。
\\
**2.2 Abstraction and Reasoning Challenge**
\\
FrançoisChollet 最近发布了一个 Kaggle 竞赛，他发布了抽象推理语料库（ARC）[12]，旨在鼓励用户创建可以解决从未接触过的推理任务的 AI 系统。希望能够开始构建更强大的 AI 系统，从而能够更好，快速地自行解决新问题，这可能有助于解决更具挑战性的现实应用，例如改善在极端和多样化环境中运行的自动驾驶汽车 。
\\
**2.3 ML and NLP Publications in 2019**
\\
Marek Rei 发布机器学习和 NLP 领域 2019 年出版数据统计[13]，分析中包括的会议是 ACL，EMNLP，NAACL，EACL，COLING，TACL，CL，CoNLL，NeurIPS，ICML，ICLR 和 AAAI。
\\
**2.4 Growing Neural Cellular Automata**
\\
`Morphogenesis`是一个自组织过程，通过该过程，蝾螈等某些生物可以再生或修复身体损伤。该过程对于扰动是鲁棒的，并且本质上是自适应的。受这种生物学现象的启发，需要更好地了解这一过程，研究人员发表了一篇题为"Growing Neural Cellular Automata"[14]的论文，该论文采用了可区分的形态发生模型，旨在复制自我修复系统的行为和特性，并希望能够制造出具有与生物生命相同的坚固性和可塑性的自我修复 machines。此外，这将有助于更好地了解再生过程本身， 许多应用注入再生医学以及社会和生物系统的建模等都将受益。
\\
\\
**2.5 Transformer attention 可视化**
\\
Hendrik Strobelt 开源项目显示了如何使用 Hugging Face 库和 d3.js 通过 Web 应用程序快速构建简单的交互式 Transformer 注意可视化[15]。
\\
\\
**2.6 SketchTransfer**
\\
SketchTransfer[16]提出了一项新任务，以测试深度神经网络在存在/不存在细节时支持不变性的能力。长期以来一直争论的是，深度网络无法推广到训练期间尚未发现的变化，人类可以相对轻松地完成某些事情，例如在观看动画片时处理丢失的视觉细节。本文讨论并发布了一个数据集，以通过提供未标记的草图图像和标记的真实图像示例来帮助研究人员仔细研究“细节不变性”问题。
\\
# 3、Tools and Datasets ⚙️
**3.1 DeepSpeed + ZeRO**
\\
Microsoft 开源了一个称为 DeepSpeed 的训练优化库，该库与 PyTorch 兼容，并且可以训练 1000 亿个参数的模型。该库侧重于训练模型的四个重要方面：规模，速度，成本和可用性。DeepSpeed 与 ZeRO[18]一起发布，ZeRO 是一种内存优化技术，可在当前 GPU 技术中实现大规模分布式深度学习，同时将吞吐量提高到当前最佳系统的三到五倍。ZeRO 允许训练任意大小的模型，这些模型可以根据共享模型状态适合聚合的可用内存。
\\
\\
**3.2 一个用于进行快速有效的 3D 深度学习研究的库**
\\
PyTorch3D[19]是用于基于 3D 的深度学习研究的开源工具包。这个 PyTorch 库旨在帮助深度学习系统中的 3D 数据支持和理解，该库由常用 3D 运算符和损失函数的快速优化实现组成， 它还带有模块化的可区分渲染器，可帮助进行复杂 3D 输入的研究并支持高质量 3D 预测。
\\
\\
**3.3 管理你的 ML 项目配置**
\\
Hydra[20]是基于 Python 的配置工具，用于更有效地管理复杂的 ML 项目。它旨在通过为 ML 项目提供功能的配置重用来帮助 PyTorch 研究人员。它提供的主要好处是它允许程序员像编写代码一样编写配置，这意味着可以轻松地覆盖配置文件。Hydra 还可以帮助自动管理 ML 项目输出的工作目录，这在需要保存和访问多个作业的多个实验结果时非常有用。
\\
**3.4 贝叶斯网络因果推理工具包**
\\
CausalNex[21]是用于“与贝叶斯网络进行因果推理”的工具包。该工具旨在将机器学习和因果推理相结合，以发现数据中的结构关系。作者还准备了关于使用 Python 库推断贝叶斯网络原因和原因的入门指南。
\\
\\
**3.5 TyDi 问答：多语言问答基准**
\\
Google AI 发布了TyDi QA[22]，它是一种多语言的数据集，可以鼓励研究人员对更多类型多样的语言进行问答，这些语言以不同的方式构建和表达含义。这样做的目的是激励研究人员在类型上距离遥远的语言（例如阿拉伯语，孟加拉语，韩语，俄语，泰卢固语和泰语）上建立更强大的模型，以便将其推广到更多种语言。
\\
\\
**3.6 Question Answering for Node.js**
\\
Hugging Face 发布了基于 DistilBERT 的问答库[23]，并继续使 NLP 更加易于访问。该模型可以使用 Node.js 在生产中运行，只需 3 行代码。该模型利用了由 Hugging Face 和 TensorFlow.js（用于将机器学习模型与 Javascript 结合使用的流行库）构建的 Tokenizer 的快速实现。
# 4、Ethics in AI 🚨
**4.1 识别文本中的 subjective bias**
\\
计算社会科学的研究员 Diyi Yang 讨论了 AI 系统如何帮助识别文本信息中的主观偏见[25]。这是涉及 AI 系统和 NLP 的重要研究领域，尤其是当我们讨论诸如新闻标题之类的文本媒体的消费时，很容易将其构架成偏向消费者，而实际上他们应该追求更加客观。从应用程序的角度来看，自动识别文本媒体中存在的主观偏见以帮助消费者更加了解他们正在消费的内容变得至关重要。此外还讨论了 AI 如何也可以保持偏见。
\\
**4.2 Artificial Intelligence, Values and Alignment**
\\
人工智能系统的兴起以及它们如何与人类价值观保持一致是涉及人工智能系统伦理学的活跃研究领域。DeepMind[26]最近发布了一篇论文，深入探讨了围绕 AI 对齐的哲学问题。该报告重点讨论了两个部分，即技术部分（即如何对从 AI 代理获得可靠结果的值进行编码）和规范性（在 AI 中进行编码的原则是正确的）以及它们之间的联系以及可以确保的部分。本文主张采用一种基于原则的 AI 对齐方法，并在信念和观点存在差异的情况下保持公平待遇。
\\
**4.3 关于审核 AI 系统**
\\
VentureBeat 报告称，Google 研究人员与其他小组合作创建了一个名为 SMACTR 的框架，该框架使工程师可以审核 AI 系统。进行这项工作的原因是为了解决目前被消费者广泛使用以供使用的 AI 系统存在的问责制差距。在这里阅读完整的报告[27]，在这里阅读完整的论文[28]。
# 5、Articles and Blog posts ✍️
**5.1 用于 NLP 系统的模型蒸馏**
\\
在NLP Highlights[29]播客的新剧集中，Thomas Wolf 和 Victor Sanh 讨论了模型蒸馏，以及如何将其用作压缩大型模型（如 BERT）以用于可扩展的实际 NLP 应用程序的可行方法。他们在他们提出的称为DistilBERT[30]的方法中对此概念进行了进一步的讨论，在该方法中，他们构建较小的模型（基于较大模型的相同体系结构），以根据该模型的输出来模仿较大模型的行为。本质上，较小的模型（学生）会尝试根据其输出分布来拟合教师的概率分布。
\\
**5.2 单词的上下文表示**
\\
最近，关于诸如 BERT 的上下文化方法成功用于处理各种复杂的 NLP 任务的讨论很多。在这篇文章中，Kawin Ethayarajh 试图回答以下问题：诸如 BERT，ELMo 和 GPT-2 之类的上下文模型及其上下文化的词表示形式是什么[31]？主题包括语境性，语境特定性的度量以及静态嵌入与语境化表示之间的比较。
\\
\\
**5.3 神经网络中的稀疏性**
\\
ML 研究人员 FrançoisLagunas 写了这篇很棒的文章，Is the future of Neural Networks Sparse?[32] 讨论了他对在神经网络模型中采用稀疏张量的乐观态度。希望采用某种形式的稀疏性来减小当前模型的大小，这些模型由于其大小和速度而在某些时候变得不切实际。由于当前模型（例如 Transformer）的庞大规模（通常依赖数十亿个参数），因此在 ML 中可能值得探讨这一概念。但是，从开发人员工具的角度来看，在 GPU 上的神经网络中支持有效稀疏性的实现细节尚不清楚，这是机器学习社区正在努力的事情。
\\
**5.4 训练你自己的语言模型**
\\
如果你想学习如何从零开始训练语言模型[33]，请查看 Hugging Face 的这份令人印象深刻且全面的教程。他们显然利用了自己的库 Transformers 和 Tokenizers 来训练模型。
\\
**5.5 分词器 Tokenizer**
\\
Cathal Horan 发表了一篇令人印象深刻且非常详细的博客文章，Tokenizers: How machines read[34]，内容涉及最新的 NLP 模型如何以及使用哪种类型的 tokenizer 来帮助机器学习算法从文本信息中学习。他还讨论并激发了为什么 tokenizer 是激动人心且重要的研究活跃领域的原因。文章甚至还展示了如何使用 SentencePiece 和 WordPiece 等标记化方法来训练自己的标记化工具。
\\
# 6、Education 🎓
**6.1 阿姆斯特丹自由大学机器学习课程**
\\
现在，你可以在线学习 2020 MLVU 机器学习课程[35]，其中包括全套幻灯片，视频和教学大纲。它旨在作为 ML 的入门，但它也包含其他与深度学习相关的主题，例如 VAE 和 GAN。
\\
**6.2 机器学习数学资源**
\\
SuzanaIlić 和东京机器学习（MLT）在使 ML 教育民主化方面一直做着惊人的工作。例如，该库Machine-Learning-Tokyo/Math_resources[36]展示了免费的在线资源集合，用于学习 ML 中使用的数学概念的基础。
\\
**6.3 深度学习入门**
\\
MIT 的“深度学习入门”课程[37]， 每周都会发布新的讲座，所有方面和视频（包括编码实验室）都将发布。
\\
**6.4 Pytorch 深度学习**
\\
Alfredo Canziani 发布了PyTorch 深度学习[38]微型课程的幻灯片和笔记本，该资源库还包含一个配套网站，其中包含对本课程中所教授概念的文字描述。
\\
**6.5 Missing Semester of Your CS**
\\
由麻省理工学院的教员发布的“Missing Semester”[39]是一门很棒的在线课程，其中包含对非开发背景的数据科学家而言可能有用的材料。它包括诸如 Shell 工具和脚本以及版本控制之类的主题。
\\
\\
**6.6 深度学习进阶**
\\
CMU 发布了“Advanced Deep Learning”[40]课程的幻灯片和教学大纲，其中包括诸如自动回归模型，生成模型以及自我监督/预测学习等主题。该课程适用于 MS 或 Ph.D， 具有 ML 高级背景的学生。
# 7、Noteworthy Mentions ⭐️
BERT-of-Theseus[41]提出了一种通过将 BERT 模型划分为原始组件来逐步替换和压缩 BERT 模型的方法。通过逐步替换和训练，还具有将模型的原始组件和压缩版本组合在一起的优势。所提出的模型优于 GLUE 基准上的其他知识提炼方法。
\\
这儿找到一份有趣的课程，称为“机器学习入门”[42]，涵盖了 ML 基础知识，监督回归，随机森林，参数调整以及许多其他基本的 ML 主题。
\\
希腊语 BERT（GreekBERT）[43]模型现在可通过 Hugging Face Transformers 库使用。
\\
Jeremy Howard[44]发表了一篇论文，描述了 fastai 深度学习库，该库被广泛用于研究并教授其深度学习开放课程， 推荐给致力于构建和改进深度学习和 ML 库的软件开发人员。
\\
Deeplearning.ai 完成了 TensorFlow 所有四个课程的发布：TensorFlow: Data and Deployment Specialization[45]。该专业的主要目的是教育开发人员如何在不同的场景中有效地部署模型，以及在训练模型时以有趣且有效的方式使用数据。
\\
Sebastian Raschka 最近发表了一篇题为《Python 中的机器学习：数据科学，机器学习和人工智能的主要发展和技术趋势[46]》的论文。本文是对机器学习工具前景的全面回顾。对于理解 ML 工程中使用的某些库和概念的各种优点而言，这是一份极好的报告。此外，还提供了有关基于 Python 的机器学习库的未来的信息。
# 本文参考资料
[1] **图灵自然语言生成（T-NLG）:** https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
\\
[2] **NVIDIA Megatron-LM:** https://github.com/NVIDIA/Megatron-LM
\\
[3] **Linguistically Informed Neural Dependency Parsing for Typologically Diverse Languages:** http://uu.diva-portal.org/smash/record.jsf?pid=diva2:1357373&dswid=7905
\\
[4] **依赖关系解析:** http://nlpprogress.com/english/dependency_parsing.html
\\
[5] **Rapid Adaptation of BERT for Information Extraction on Domain-Specific Business Documents:** https://arxiv.org/abs/2002.01861
\\
[6] **question understanding benchmark:** https://arxiv.org/abs/2001.11770v1
\\
[7] **有趣的工作:** https://ai.facebook.com/blog/using-radioactive-data-to-detect-if-a-data-set-was-used-for-training/
\\
[8] **完整论文:** https://arxiv.org/pdf/2002.00937.pdf
\\
[9] **REALM: Retrieval-Augmented Language Model Pre-Training:** https://kentonl.com/pub/gltpc.2020.pdf
\\
[10] **change.org:** https://www.change.org/p/organizers-of-data-science-and-machine-learning-conferences-neurips-icml-aistats-iclr-uai-allow-remote-paper-poster-presentations-at-conferences[11]**新博客:** https://yoshuabengio.org/2020/02/10/fusce-risus/
\\
[12] **抽象推理语料库（ARC）:** https://www.kaggle.com/c/abstraction-and-reasoning-challenge/overview
\\
[13] **机器学习和 NLP 领域 2019 年出版数据统计:** https://www.marekrei.com/blog/ml-and-nlp-publications-in-2019/
\\
[14] **Growing Neural Cellular Automata:** https://distill.pub/2020/growing-ca/[15]**交互式 Transformer 注意可视化:** https://github.com/SIDN-IAP/attnvis
\\
[16] **SketchTransfer:** https://arxiv.org/pdf/1912.11570.pdf
\\
[18] **DeepSpeed 与 ZeRO:** https://www.microsoft.com/en-us/research/blog/zero-deepspeed-new-system-optimizations-enable-training-models-with-over-100-billion-parameters/
\\
[19] **PyTorch3D:** https://ai.facebook.com/blog/-introducing-pytorch3d-an-open-source-library-for-3d-deep-learning/
\\
[20] **Hydra:** https://hydra.cc/
\\
[21] **CausalNex:** https://causalnex.readthedocs.io/en/latest/01_introduction/01_introduction.html
\\
[22] **TyDi QA:** https://ai.googleblog.com/2020/02/tydi-qa-multilingual-question-answering.html
\\
[23] **问答库:** https://github.com/huggingface/node-question-answering
\\
[25] **识别文本信息中的主观偏见:** https://podcasts.apple.com/us/podcast/will-ai-help-identify-bias-or-perpetuate-it-with-diyi-yang/id1435564422?i=1000464141922
\\
[26] **DeepMind:** https://deepmind.com/research/publications/Artificial-Intelligence-Values-and-Alignment[27]**完整的报告:** https://venturebeat.com/2020/01/30/google-researchers-release-audit-framework-to-close-ai-accountability-gap/
\\
[28] **完整的论文:** https://dl.acm.org/doi/abs/10.1145/3351095.3372873
\\
[29] **NLP Highlights:** https://soundcloud.com/nlp-highlights/104-model-distillation-with-victor-sanh-and-thomas-wolf
\\
[30] **DistilBERT:** https://arxiv.org/abs/1910.01108
\\
[31] **诸如 BERT，ELMo 和 GPT-2 之类的上下文模型及其上下文化的词表示形式是什么:** https://kawine.github.io/blog/nlp/2020/02/03/contextual.html
\\
[32] **Is the future of Neural Networks Sparse?:** https://medium.com/huggingface/is-the-future-of-neural-networks-sparse-an-introduction-1-n-d03923ecbd70
\\
[33] **从零开始训练语言模型:** https://huggingface.co/blog/how-to-train
\\
[34] **Tokenizers: How machines read:** https://blog.floydhub.com/tokenization-nlp/
\\
[35] **MLVU 机器学习课程:** https://mlvu.github.io/
\\
[36] **Machine-Learning-Tokyo/Math_resources:** https://github.com/Machine-Learning-Tokyo/Math_resources
\\
[37] **MIT 的“深度学习入门”课程:** http://introtodeeplearning.com/
\\
[38] **PyTorch 深度学习:** https://atcold.github.io/pytorch-Deep-Learning-Minicourse/
\\
[39] **“Missing Semester”:** https://missing.csail.mit.edu/
\\
[40] **“Advanced Deep Learning”:** https://andrejristeski.github.io/10707-S20/syllabus.html
\\
[41] **BERT-of-Theseus:** http://xxx.itp.ac.cn/abs/2002.02925
\\
[42] **“机器学习入门”:** https://compstat-lmu.github.io/lecture_i2ml/index.html
\\
[43] **希腊语 BERT（GreekBERT）:** https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1
\\
[44] **Jeremy Howard:** https://arxiv.org/abs/2002.04688
\\
[45] **TensorFlow: Data and Deployment Specialization:** https://www.coursera.org/specializations/tensorflow-data-and-deployment
\\
[46] **Python 中的机器学习：数据科学，机器学习和人工智能的主要发展和技术趋势:** https://arxiv.org/abs/2002.04803
---
## File: posts/_posts/2020-02-20-NLP简报_Tokenizers,_TensorFlow_2_1,_TextVectorization.md
---
layout: post
title: "NLP简报 #1 [ZH]: Tokenizers, TensorFlow 2.1, TextVectorization, TorchIO, NLP Shortfalls,…"
author: billy_rick
modified:
comments: true
excerpt: ""
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_1.png
---
\\
*Thanks to* [*Kaiyuan*](https://blog.csdn.net/Kaiyuan_sjtu) *(WeChat: NewBeeNLP) for this great translation.*
\\
欢迎来到船新栏目**「NLP 简报」**，本新闻简报的目的是让你不必花费太多时间就可以了解与 NLP 和 ML 有关的一些有趣和最新的故事。
\\
**如果想让自己有趣的研究/项目出现在 NLP 简报中，随时在公众号后台留言联系我**
\\
下面来看看第一期的内容
- **1、Publications 📙**
- 1.1 用于乳腺癌筛查的AI系统
- 1.2 信息抽取
- 1.3 Improved recommendation
- **2、Creativity and Society 🎨**
- 2.1 AI就业
- **3、Tools and Datasets ⚙️**
- 3.1 一个极速分词器
- 3.2 用于搜索的ML&NLP
- 3.3 医学图像分析
- **4、Ethics in AI 🚨**
- 4.1 ML社区的欺诈行为
- 4.2 机器翻译中的性别偏见
- 4.3 ML偏差与公正性
- **5、Articles and Blog posts ✍️**
- 5.1 NLP shortfalls
- 5.2 NLP和ML2019年亮点
- **6、Education 🎓**
- 6.1 Democratizing AI education
- 6.2 Top NLP and ML Books
- 6.3 使用核方法的机器学习
- **7、Noteworthy Mentions ⭐️**
# 1、Publications 📙
**1.1 用于乳腺癌筛查的 AI 系统**
\\
DeepMind 在 Nature 杂志上发表了一篇名为“International evaluation of an AI system for breast cancer screening[1]”的新论文。作者说，这项工作是关于在乳腺癌筛查方面超越人类专家的 AI 系统的评估。当前的 AI 系统是否真的可以实现这一点尚有争议，并且对于这种类型的系统及其评估方式一直存在批评。
\\
**1.2 信息抽取**
\\
Pankaj Gupta 公开发布了他的博士学位论文，题目为“Neural Information Extraction From Natural Language Text[2]”。主要讨论如何使用基于神经的方法有效地从自然语言文本中提取语义关系， 此类研究工作旨在促进构建结构化的知识库，该知识库可用于一系列下游 NLP 应用程序，例如 Web 搜索，问题解答以及其他任务。
\\
**1.3 Improved recommendations**
\\
MIT 和 IBM 的研究人员基于三种广泛使用的文本分析工具（主题建模，单词嵌入和最佳传输）的组合，开发了一种用于分类，显示和搜索相关文档的方法（于去年在 NeurIPS 上发布）。该方法还为文档排序提供了有希望的结果。此类方法适用于需要对大规模数据（例如搜索和推荐系统）进行改进和更快建议的各种场景和应用。
# 2、Creativity and Society 🎨
**2.1 AI 就业**
\\
Stanford 2019 年 AI 指数报告[3]表明，对 AI 从业者的需求更多。但是，与 AI 相关的工作有很多方面，例如职业转变和面试仍然没有适当定义。
\\
在博客"Shifting Careers to Autonomous Vehicles[4]"中，Vladimir Iglovivok 详细介绍了他的职业生涯和 ML 冒险，从构建传统的推荐系统到构建壮观的计算机视觉模型（赢得了 Kaggle 竞赛）。他现在在 Lyft 从事自动驾驶汽车的工作，但是到达那里的旅程并不容易。
\\
如果 n 真的对 AI 事业感兴趣并且很认真，Andrew Ng 的公司 deeplearning.ai 成立了 Workera，该公司专门致力于帮助数据科学家和机器学习工程师从事 AI 事业。可以在此处获取关于Workra 的官方报告[5]。
# 3、Tools and Datasets ⚙️
**3.1 一个极速分词器**
\\
Hugging Face 是一家的 NLP 初创公司，拥有开源的 Tokenizers，这是一种可在现代 NLP pipeline 中使用的超快速的分词器，可以查看Tokenizers GitHub 库[6]以获取有关如何使用 Tokenizer 的文档。
\\
\\
TensorFlow 2.1 合并了一个新的TextVectorization 层[7]，你可以轻松处理原始字符串并有效地执行文本 normalization，tokenization，n-gram 生成和词汇索引。点击查看Chollet 的 Colab 笔记[8]本，演示如何使用该功能进行端到端文本分类。
\\
**3.2 用于搜索的 ML&NLP**
\\
去年，NLP 取得了巨大进步，其中一个领域是一系列改进和新的研究方向。搜索可能是可能从迁移学习 NLP 中受益的那些领域之一。
\\
尽管搜索属于信息检索领域，但仍有机会构建使用现代 NLP 技术（例如来自基于BERT[9]的基于变压器的模型的上下文表示）来改进语义搜索的搜索引擎。Google 在几个月前发布了一篇博客文章，讨论了他们如何利用 BERT 模型来改善和理解搜索[10]。
\\
如果您对如何将上下文化表示形式应用于使用 Elasticsearch 和 TensorFlow 等开放式搜索技术的搜索感到好奇，则可以查看"Elasticsearch meets BERT"[11]或"Building a Search Engine with BERT and TensorFlow"[12]。
\\
**3.3 医学图像分析**
\\
TorchIO[13]是基于流行的深度学习库 PyTorch 的 Python 软件包。TorchIO 提供的功能可轻松高效地读取和采样 3D 医学图像。功能包括用于数据扩充和预处理的空间变换。
\\
# 4、Ethics in AI 🚨
**4.1 ML 社区的欺诈行为**
\\
Kaggle 比赛的第一名优胜者因欺诈活动而被取消参赛资格，其队伍使用了聪明但不负责任和不可接受的策略来赢得比赛的第一名。原文"PetFinder.my Contest: 1st Place Winner Disqualified[14]"重点介绍了机器学习社区想要缓解的许多严重且无法接受的行为，正确和道德地使用机器学习技术是前进的唯一方法。
\\
**4.2 机器翻译中的性别偏见**
\\
关于机器翻译系统是否反映性别偏见的主题，一组研究人员发表了这篇出色的论文，"Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate[15]"，提出了使用 Google 翻译的案例研究。作者声称的一项发现是，Google 翻译“表现出强烈的男性违约倾向，特别是在与性别分布失衡有关的领域，例如 STEM 工作。”
\\
**4.3 ML 偏差与公正性**
\\
如果您想让所有人都了解 AI 伦理和公平，那么这是一个由Timnit Gebru[16]主持，由 TWIML 主持的不错的播客。
\\
Timnit 是 ML 公平性方面的杰出研究者，他与 Eun Seo Jo 一起发表了一篇论文，"Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning"[17]他们确定了档案中文档收集实践的五种关键方法，这些方法可以为社会文化 ML 中的数据收集提供更可靠的方法。这可能会导致跨学科合作研究获得更系统的数据收集方法。
\\
Sina Fazelpour 和 Zachary Lipton 最近发表了一篇论文，"fairness-non-ideal-fazelpour-lipton-2020[18]"，他们认为，由于我们非理想世界的产生方式的性质，基于理想思维的公平 ML 可能会导致误导政策和干预措施。实际上，他们的分析表明“提出的公平 ML 算法的缺点反映了理想方法所面临的广泛问题。”
# 5、Articles and Blog posts ✍️
**5.1 NLP shortfalls**
\\
Benjamin Heinzerling 在 The Gradient 中发表了一篇有趣的文章，讨论了NLP 不足的领域[19]，例如论点理解和常识推理。本杰明参考了 Nivin＆Kao 的最新论文，"Probing Neural Network Comprehension of Natural Language Arguments[20]"，该论文挑战和质疑了转移学习和语言模型对高级自然语言理解的能力。
\\
**5.2 NLP 和 ML2019 年亮点**
\\
对于新的一年，报告"NLP Year in Review — 2019[21]"，记录了 2019 年的一些最有趣的 NLP 和 ML 亮点。
\\
塞巴斯蒂安·鲁德（Sebastian Ruder）最近还写了一篇精彩而详尽的博客文章[22]，介绍了十大关于 ML 和 NLP 的研究方向，他认为这很有影响力 在 2019 年。列表中包括诸如通用无监督预训练，应用于科学的 ML 和 NLP，增强预训练模型，高效和远程 Transformers 等主题。
\\
Google AI 研究会发布他们一年来进行的研究的摘要以及他们正在关注的未来研究方向，"Google Research: Looking Back at 2019, and Forward to 2020 and Beyond[23]"。
# 6、Education 🎓
**6.1 Democratizing AI education**
\\
为了使 AI 教育民主化并向大众普及 AI 技术的影响，赫尔辛基大学与 Reaktor 合作发布了涵盖 AI 基础知识的精彩免费课程。受欢迎的课程称为“Elements of AI[24]”，包括诸如 AI 伦理学，AI 哲学，神经网络，朴素贝叶斯规则等主题，以及其他基础主题。
\\
Stanford CS224N 再次推出了流行的“Natural Language Processing with Deep Learning[25]”课程。该课程于今年 1 月 7 日正式开始，因此，如果您想学习该课程，请访问其网站以获取完整的课程提纲，幻灯片，视频，阅读建议等。
\\
**6.2 Top NLP and ML Books**
\\
关于一些NLP 和 ML 领域书籍推荐[26]。
\\
**6.3 使用核方法的机器学习**
\\
诸如 PCA 和 K-means 之类的核方法已经存在了很长一段时间，这是因为它们已成功应用于各种应用，例如图形和生物序列。查看这套涵盖了各种Kernel Methods[27]及其内部工作原理的综合幻灯片。这也是一个由 Francis Bach 维护很棒的博客，"Are all kernels cursed?[28]"，讨论了内核方法和其他机器学习主题。
# 7、Notable Mentions ⭐️
另外还有一些有趣的研究/项目：
- John Langford 的博客[29]，讨论机器学习理论；
- 许多行业的面向机器学习的技术已经使用梯度增强机器多年了。看看这篇文章[30]，其中介绍了一个用于应用梯度增强的库 XGBoost。
- 如果你对学习如何设计和构建基于机器学习的应用程序并将其投入生产感兴趣，那么Emmanuel Ameisen[31]的书籍可以帮助你。
\\
**本文参考资料**
\\
[1] **International evaluation of an AI system for breast cancer screening:** https://www.nature.com/articles/s41586-019-1799-6
\\
[2] **Neural Information Extraction From Natural Language Text:** https://www.researchgate.net/publication/336739252_PhD_Thesis_Neural_Information_Extraction_From_Natural_Language_Text
\\
[3] **Stanford 2019 年 AI 指数报告:** https://hai.stanford.edu/sites/g/files/sbiybj10986/f/ai_index_2019_report.pdf
\\
[4] **Shifting Careers to Autonomous Vehicles:** https://towardsdatascience.com/how-i-found-my-current-job-3fb22e511a1f
\\
[5] **Workra 的官方报告:** https://workera.ai/candidates/report
\\
[6] **Tokenizers GitHub 库:** https://github.com/huggingface/tokenizers
\\
[7] **TextVectorization 层:** https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization
\\
[8] **Chollet 的 Colab 笔记:** https://colab.research.google.com/drive/1RvCnR7h0_l4Ekn5vINWToI9TNJdpUZB3
\\
[9] **BERT:** https://arxiv.org/abs/1810.04805
\\
[10] **利用 BERT 模型来改善和理解搜索:** https://www.blog.google/products/search/search-language-understanding-bert/
\\
[11] **Elasticsearch meets BERT:** https://towardsdatascience.com/elasticsearch-meets-bert-building-search-engine-with-elasticsearch-and-bert-9e74bf5b4cf2
\\
[12] **Building a Search Engine with BERT and TensorFlow:** https://towardsdatascience.com/building-a-search-engine-with-bert-and-tensorflow-c6fdc0186c8a
\\
[13] **TorchIO:** https://github.com/fepegar/torchio
\\
[14] **PetFinder.my Contest: 1st Place Winner Disqualified:** https://www.kaggle.com/c/petfinder-adoption-prediction/discussion/125436
\\
[15] **Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate:** https://arxiv.org/abs/1809.02208
\\
[16] **Timnit Gebru:** https://twimlai.com/twiml-talk-336-trends-in-fairness-and-ai-ethics-with-timnit-gebru/
\\
[17] **Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning:** https://arxiv.org/abs/1912.10389
\\
[18] **fairness-non-ideal-fazelpour-lipton-2020:** http://zacklipton.com/media/papers/fairness-non-ideal-fazelpour-lipton-2020.pdf
\\
[19] **NLP 不足的领域:** https://thegradient.pub/nlps-clever-hans-moment-has-arrived/
\\
[20] **Probing Neural Network Comprehension of Natural Language Arguments:** https://www.aclweb.org/anthology/P19-1459/
\\
[21] **NLP Year in Review — 2019:** https://medium.com/dair-ai/nlp-year-in-review-2019-fb8d523bcb19
\\
[22] **精彩而详尽的博客文章:** https://ruder.io/research-highlights-2019/
\\
[23] **Google Research: Looking Back at 2019, and Forward to 2020 and Beyond:** https://ai.googleblog.com/2020/01/google-research-looking-back-at-2019.html
\\
[24] **Elements of AI:** https://www.elementsofai.com/
\\
[25] **Natural Language Processing with Deep Learning:** http://web.stanford.edu/class/cs224n/
\\
[26] **NLP 和 ML 领域书籍推荐:** https://twitter.com/omarsar0/status/1214547402838986754?s=20
\\
[27] **Kernel Methods:** http://members.cbio.mines-paristech.fr/~jvert/svn/kernelcourse/slides/master2017/master2017.pdf
\\
[28] **Are all kernels cursed?:** https://francisbach.com/cursed-kernels/
\\
[29] **John Langford 的博客:** https://hunch.net/
\\
[30] **这篇文章:** https://opendatascience.com/xgboost-enhancement-over-gradient-boosting-machines/?utm_campaign=Learning%20Posts&utm_content=111061559&utm_medium=social&utm_source=twitter&hss_channel=tw-3018841323
\\
[31] **Emmanuel Ameisen:** https://www.amazon.com/Building-Machine-Learning-Powered-Applications/dp/149204511X/
---
## File: posts/_posts/2020-02-21-Fundamentals-of-NLP-Chapter-1.md
---
layout: post
title: "Fundamentals of NLP (Chapter 1): Tokenization, Lemmatization, Stemming, and Sentence Segmentation"
author: billy_rick
modified:
comments: true
excerpt: "In this first chapter of the Fundamentals of NLP series, we will learn about lemmatization, stemming, tokenization, and sentence segmentation."
tags: [nlp_fundamentals]
image:
thumb: chapter1.png
---
\\
Natural language processing (NLP) has made substantial advances in the past few years due to the success of [modern techniques](https://nlpoverview.com/) that are based on [deep learning](https://en.wikipedia.org/wiki/Deep_learning). With the rise of the popularity of NLP and the availability of different forms of large-scale data, it is now even more imperative to understand the inner workings of NLP techniques and concepts, from first principles, as they find their way into real-world usage and applications that affect society at large. Building intuitions and having a solid grasp of concepts are both important for coming up with innovative techniques, improving research, and building safe, human-centered AI and NLP technologies.
\\
In this first chapter, which is part of a series called [Fundamentals of NLP](https://github.com/dair-ai/nlp_fundamentals), we will learn about some of the most important basic concepts that power NLP techniques used for research and building real-world applications. Some of these techniques include lemmatization, stemming, tokenization, and sentence segmentation. These are all important techniques to train efficient and effective NLP models. Along the way, we will also cover best practices and common mistakes to avoid when training and building NLP models. We also provide some exercises for you to keep practicing and exploring some ideas.
\\
In every chapter, we will introduce the theoretical aspect and motivation of each concept covered. Then we will obtain hands-on experience by using bootstrap methods, industry-standard tools, and other open-source libraries to implement the different techniques. Along the way, we will also cover best practices, share important references, point out common mistakes to avoid when training and building NLP models, and discuss what lies ahead.
\\
The project will be maintained on [GitHub](https://github.com/dair-ai/nlp_fundamentals).
\\
Get access to the first chapter at this [Colab notebook](https://colab.research.google.com/drive/18ZnEnXKLQkkJoBXMZR2rspkWSm9EiDuZ).
---
## File: posts/_posts/2020-02-23-Boletín_informativo_NLP_GPT-2_Explicado,_Entendie.md
---
layout: post
title: "Boletín informativo NLP #5 [ES]: GPT-2 Explicado, Entendiendo ‘Self-Distillation’, Haiku, GANILLA, Sparkwiki, Ética en el NLP, Torchmeta,..."
author: naraquev
modified:
comments: true
excerpt: ""
tags: [nlp_newsletter]
image:
thumb: nlp_newsletter_5.png
---
\\
Post Original en Ingles: [https://dair.ai/NLP_Newsletter_The_Annotated_GPT-2,_Understanding/](https://dair.ai/NLP_Newsletter_The_Annotated_GPT-2,_Understanding/)
\\
Primero que todo, no puedo agradecerles lo suficiente a todos por el increible animo y soporte para continuar con el boletín informativo de NLP. Este esfuerzo requiere una investigación y edición tediosa, la cual encuentro gratificante y útil para proveer el mejor contenido. Espero que lo estén disfrutando tanto como yo.
\\
[Suscríbete](https://dair.ai/newsletter/) al Boletín Informativo de NLP para recibir futuras ediciones en tu email. Este boletin lo desarrolla Elvis Saravia, editor de dair.ai
# Publicaciones📙
***Un entendimiento teórico del self-distillation***
\\
En el contexto del Deep Learning, self-distillation es el proceso de transferir conocimiento de una arquitectura a otra arquitectura idéntica. Las predicciones del modelo original son alimentadas como valores objetivo al otro modelo mientras se entrena. Además de tener propiedades deseables (como reducir el tamaño del modelo) resultados empíricos demuestran que esta aproximación trabaja bastante bien en sets de datos del tipo held-out. Un grupo de investigadores recientemente publicó una investigación que provee un análisis teórico con foco en entender mejor qué está pasando en el proceso de destilación del conocimiento y por que es efectivo. Los resultados muestran que unas pocas rondas de self-distillation amplifica la regularización ([limitando progresivamente el número de funciones básicas que representan la solución](https://twitter.com/TheGradient/status/1228132843630387201?s=20)) lo que tiende a reducir el over-fitting. (Lee la investigación completa [aquí](https://arxiv.org/abs/2002.05715))
\\
\\
*Traducción: Figura 1. Ilustración esquemática del proceso de self-distillation por dos iteraciones. Fuente: [https://arxiv.org/abs/2002.05715](https://arxiv.org/abs/2002.05715)*
\\
***Los 2010: La década del Deep Learning / Mirada al 2020***
\\
[Jürgen Schmidhuber,](http://people.idsia.ch/~juergen/) un pionero en la inteligencia artificial, publicó recientemente un [artículo](http://people.idsia.ch/~juergen/2010s-our-decade-of-deep-learning.html) titulado “Foco en proveer un resumen histórico del Deep Learning desde el 2010”. Algunos de los temas incluyen LSTMs, redes neuronales feedforward, GANs, DeepRL, Meta-Learning, World Models, distillings NNs, Attention Learning, etc. El artículo concluye con una mirada al 2020, donde llama la atención a temas como la privacidad y los mercados de datos.
\\
***Utilizando Redes Neuronales para resolver ecuaciones matemáticas avanzadas.***
\\
Investigadores de Facebook AI publicaron una [investigación](https://arxiv.org/abs/1912.01412) que dice proponer un modelo entrenado en problemas matemáticos y sus soluciones para aprender a predecir posibles soluciones en problemas cómo resolver integrales matematicas. El acercamiento está basado en un novedoso framework similar al usado en neural machine translation donde una expresión matemática es representada como una especie de lenguaje y la solución es tratada como su traducción. Por esto, en vez de que el resultado sea una traducción, es la solución al problema matematico. Con esto, los investigadores concluyen que las redes neuronales profundas no solo son buenas para el razonamiento simbólico sino también para tareas más diversas.
\\
*Ecuaciones incluidas como entrada con su correspondiente solución. [Fuente](https://ai.facebook.com/blog/using-neural-networks-to-solve-advanced-mathematics-equations/)*
# Creatividad y Sociedad 🎨
***IA para el descubrimiento científico***
\\
Mattew Hutson [reporta](https://www.sciencemag.org/news/2020/02/models-galaxies-atoms-simple-ai-shortcuts-speed-simulations-billions-times) cómo la inteligencia artificial (IA) puede ser usada para producir emuladores con la capacidad de modelar fenómenos naturales complejos. Estos modelos pueden a su vez apuntar a diferentes tipos de descubrimientos científicos. El desafío al construir estos emuladores es que necesitan una gran cantidad de datos y una búsqueda extensiva de parámetros. Una [investigación](https://arxiv.org/abs/2001.08055) reciente propone una técnica llamada DENSE, basada en [neural architecture search](https://en.wikipedia.org/wiki/Neural_architecture_search) para construir emuladores precisos utilizando una cantidad de datos de entrenamiento limitada. Los investigadores hicieron distintas pruebas construyendo simuladores para caso como astrofísica, ciencia climática, energía de fusión, etc.
\\
***Mejorando la traducción imagen-a-ilustración***
\\
GANILLA es una aproximación que propone el uso de las GANs para mejorar la transferencia de estilo y contenido en la tarea de convertir una imagen en una ilustración. En particular, un modelo para [imagen-a-ilustración](https://paperswithcode.com/task/image-to-image-translation) es propuesto (con una red generadora mejorada) y evaluado basado en un nuevo framework para evaluación cuantitativa que considera tanto el contenido como el estilo de la imagen generada. La novedad de esta investigación es en la red generadora propuesta que considera un balance entre estilo y contenido que anteriores intentos fallaron en alcanzar. El código y modelo pre entrenado están disponibles en [línea](https://github.com/giddyyupp/ganilla). [Aquí](https://arxiv.org/abs/2002.05638) se puede encontrar la investigación completa.
\\
*Traducción: Ejemplos de salida de CycleGAN, DualGAN, y nuestro método (GANILLA) utilizando diferentes estilos. CycleGAN y GANILLA generan imágenes en un estilo de ilustración, pero DualGAN falla en transferir el estilo. Sin embargo, CycleGAN falle en preservar el contenido de la imagen original, por ejemplo, aleatoriamente coloca caras de animales en el aire. Nuestro método preserva contenido así como también transfiere el estilo de la imagen original.*
\\
***Andrew Ng habla sobre su interés en self-supervised learning***
\\
Andrew Ng, fundador de [deeplearning.ai](http://deeplearning.ai/), participa en el podcast Artificial Inteligence de Lex Friedman para [hablar](https://www.youtube.com/watch?v=0jspaMLxBig) de diferentes temas como sus comienzos en el ML, el futuro del AI y la educación, recomendaciones de uso para ML, sus metas personales y a cuáles técnicas de ML prestar atención en el 2020.
\\
Andrew tambien explico por que está muy emocionado sobre el self-supervised representation learning. [Self-supervised learning](https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html) se trata sobre enfocar un problema de aprendizaje que intenta obtener la supervisión de los datos en sí mismos para hacer uso de grandes cantidades de datos no clasificados, escenario más común que datos limpios y clasificados. Las representaciones aprendidas, opuesto al rendimiento de la tarea, son importantes y pueden ser usadas en tareas más adelante, similar a lo que está siendo usado en modelos de lenguaje natural como [BERT](https://lilianweng.github.io/lil-log/2019/01/31/generalized-language-models.html#bert).
\\
Hay mucho interés en usar el self-supervised learning para aprender representaciones visuales generales que hagan al modelo más preciso en situaciones de bajos recursos. Por ejemplo, un nuevo método llamado [SimCLR](https://arxiv.org/abs/2002.05709) (Dirigido por Geoffrey Hinton) propone un framework para el *constrastive self-supervised learning* de representaciones visuales para mejorar el resultado de la clasificación de imágenes en diferentes contextos como el *transfer learning* y el *semi-supervised learning*.
\\
*Traducción: ImageNet top-1 accuracy de clasificadores lineales entrenados en representaciones aprendidas con diferentes métodos de auto-aprendizaje (pre entrenados en ImageNet). La cruz gris indica un modelo ResNet-50 supervisado. Nuestro método, SimCLR, es mostrado en negrita.*
# Herramientas y Set de Datos ⚙️
***Librerías JAX***
\\
[JAX](https://github.com/google/jax) es una nueva librería que combina NumPy y diferenciación automática para realizar investigación de alto rendimiento en ML. Para simplificar procesos para construir redes neuronales usando JAX, DeepMind lanzó [Haiku](https://github.com/deepmind/dm-haiku) y [RLax](https://github.com/deepmind/rlax). RLAx simplifica la implementación de agentes de reinforcement learning y Haiku simplifica la construcción de redes neuronales utilizando paradigmas familiares de programación orientada a objetos.
\\
***Una herramienta para procesar datos de Wikipedia***
\\
[Spakwiki](https://github.com/epfl-lts2/sparkwiki) es una herramienta para procesar datos de Wikipedia. Este lanzamiento es parte de muchos esfuerzos para permitir análisis interesantes de comportamiento como [capturar tendencias y sesgos de lenguaje en ediciones de distinto idioma en Wikipedia](https://arxiv.org/abs/2002.06885). Los autores descubrieron que independientemente del lenguaje, el comportamiento de búsqueda de los usuarios de Wikipedia es muy parecido en categorías como películas, música y deportes pero que las diferencias se vuelven más aparentes con eventos locales y particularidades de cultura.
\\
\\
***Tokenizadores, caso DistilBERT y modelos***
\\
Una [nueva entrega de Transformers](https://github.com/huggingface/transformers/releases/tag/v2.5.0) de Hugging Face ahora incluye la integración de su rápida librería de tokenización que intenta mejorar el tiempo de ejecución de modelos como BERT, RoBERTa, GPT2, y otro modelos construidos por la comunidad.
# Ética en la Inteligencia Artificial 🚨
***Consideraciones éticas para modelos de NLP y ML.***
\\
En un nuevo [episodio](https://soundcloud.com/nlp-highlights/106-ethical-considerations-in-nlp-research-emily-bender) de [NLP Highlights](https://soundcloud.com/nlp-highlights), Emily Bender habla sobre algunas consideraciones éticas cuando se desarrollan modelos de Procesamiento de Lenguaje Natural y tecnologías tanto en el contexto de la academia como de industria. Algunos de los tópicos discutidos incluyen consideraciones éticas cuando se diseñan tareas, recolección de datos, y eventualmente publicación de resultados.
\\
Adicionalmente a todas las consideraciones mencionadas anteriormente, una preocupación que siempre se discute en la comunidad de AI es enfocarse demasiado en optimizar una métrica, lo que va en contra de los cimientos de lo que el campo trata de alcanzar. Rachel Thomas y David Uminsky discuten cómo esto puede salir mal a través de un [análisis](https://arxiv.org/abs/2002.08512) con diferentes casos de uso. Además, proponen un framework simple para mitigar el problema que involucra el uso y combinación de múltiples métricas, seguido del involucramiento directo de los usuarios afectados por la tecnología que se está desarrollando.
# Artículos y Blogs ✍️
***GPT-2 Explicado***
\\
Aman Arora recientemente publicó un artículo excepcional titulado “[**El GPT-2 Explicado**](https://amaarora.github.io/2020/02/18/annotatedGPT2.html)” explicando el funcionamiento interno del modelo basado en la técnica del Transformer llamado GPT-2. Su aproximación fue inspirada por el artículo [El Transformer Explicado](https://nlp.seas.harvard.edu/2018/04/03/attention.html) que tomó una aproximación a explicar las partes más importantes del modelo siguiendo ejemplos fáciles de seguir y código comentado. Amant realizó un gran esfuerzo para re implementar GPT-2 de OpenAI utilizando PyTorch y la librería de Transformers de Hugging Face. Es un trabajo brillante.
\\
*[Fuente](https://amaarora.github.io/2020/02/18/annotatedGPT2.html)*
\\
***Más allá de BERT?***
\\
[Opinión](https://towardsdatascience.com/beyond-bert-6f51a8bc5ce1) interesante de Sergi Castella en que hay más allá de BERT. El tema principal incluye mejorar métricas, como la librería Transformer de Hugging Face empodera la investigación, set de datos interesantes para revisar, utilización de modelos, etc.
\\
***Operador para comprimir Matrices***
\\
El blog the TensorFlow publico un [articulo](https://blog.tensorflow.org/2020/02/matrix-compression-operator-tensorflow.html?linkId=82298016) donde explican las técnicas para compresión de matrices y su importancia en un modelo de red neuronal. La compresión de matrices puede ayudar a construir modelos más eficientes y reducidos que pueden ser incorporados en dispositivos más pequeños como teléfonos y asistentes del hogar. Enfocarse en la comprensión por métodos como low-rank-approximation y quantization significa que no debemos comprometer la calidad del modelo
\\
*[Fuente](https://blog.tensorflow.org/2020/02/matrix-compression-operator-tensorflow.html?linkId=82298016)*
# Educación🎓
***Fundamentos del NLP***
\\
Estoy emocionado de liberar un borrador del capítulo 1 de mi nueva serie llamado [Fundamentos del NLP](https://medium.com/dair-ai/fundamentals-of-nlp-chapter-1-tokenization-lemmatization-stemming-and-sentence-segmentation-b362c5d07684). Enseña conceptos comenzando desde lo más básico, compartiendo buenas prácticas, referencias importantes, errores comunes a evitar, y cuál es el futuro del NLP. Un [Colab Notebook](https://colab.research.google.com/drive/18ZnEnXKLQkkJoBXMZR2rspkWSm9EiDuZ) está incluido y el proyecto será mantenido [aquí](https://github.com/dair-ai/nlp_fundamentals)
\\
\\
***[En línea] Revisión/Discusión: Parte 1 Sesión de lectura de Fundamentos Matemáticos***
\\
Machine Learning Tokio está organizando una discusión en línea para revisar los capítulos que fueron cubiertos en su más reciente sesión de estudio. El grupo ha estudiado previamente capítulos del libro [Matemáticas para el Machine Learning](https://mml-book.github.io/) escrito por Marc Peter Deisenroth, A Aldo Faisal, y Cheng Soon Ong. El [evento](https://www.meetup.com/Machine-Learning-Tokyo/events/268817313/) está pautado para el 8 de Marzo de 2020.
\\
***Libros Recomendados***
\\
En un segmento previo discutimos la importancia de la compresión de matrices para construir modelos de ML compactos. Si estás interesado en aprender más sobre como construir redes neuronales más pequeñas para sistemas embebidos te recomiendo este gran libro llamado [TinyML](https://tinymlbook.com/?linkId=82595412) por Pete Warden y Daniel Situnayake.
\\
*[Fuente](https://www.amazon.com/TinyML-Learning-TensorFlow-Ultra-Low-Micro-Controllers/dp/1492052043)*
\\
Otro libro interesante a tener en cuenta es “[Deep Learning para programadores con fastai y pyTorch: Aplicaciones AI sin un PhD](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527)” por Jeremy Howard y Sylvain Gugger. El libro trata de proveer la matemática necesaria para construir y entrenar modelos que resuelvan tareas en el área de visión por computador y Entendimiento natural de texto.
\\
*[Fuente](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527)*
# Menciones a considerar ⭐️
Puedes acceder a las versiones anteriores de este boletín [aquí](https://medium.com/dair-ai/nlp-newsletter-pytorch3d-deepspeed-turing-nlg-question-answering-benchmarks-hydra-sparse-322f018ee096).
\\
[Torchmeta](https://arxiv.org/abs/1909.06576) es una librería que permite el uso de data loaders para la investigación en meta-learning. El creador es Tristan Deleu.
\\
Manuel Tonneau escribió una [pieza](https://creatext.ai/blog-posts/machine-text-writing-gpt2-beam-search?utm_medium=newsletter) ofreciendo una mirada más detallada a alguna de la maquinaria involucrada en el modelamiento del lenguaje. Algunos tópicos incluyen búsquedas greedy y beam, así como también nucleus sampling.
\\
El MIT [liberó](http://introtodeeplearning.com/) el programa de estudio completo del curso titulado “Introducción al Deep Learning”, incluye videos de las clases que ya se dictaron. Están apuntando a liberar videos y presentaciones todas las semanas.
\\
Aprende a entrenar un modelo para reconocimiento de entidades nombradas (NER) utilizando una aproximación vía Transformers en menos de [300 líneas de código](https://github.com/huggingface/transformers/blob/master/examples/ner/run_pl_ner.py). Puedes encontrar el Google Colab que lo acompaña [aquí](https://colab.research.google.com/drive/184LPlygvdGGR64hgQl3ztqzZJu8MmITn).