## 1. Project Overview & Quickstart (dmlc/LICENSE) # LICENSE Open-source repository dmlc/LICENSE ### Repository Details - **Repository:** [dmlc/LICENSE](https://github.com/dmlc/LICENSE) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (dmlc/dmlc.github.io) ## File: README.md How to use: First put the markdown format post in `_posts/` in format `year-month-day-title.markdown`, and then commit Preview in local: ``` $ gem install jekyll pygments.rb redcarpet $ jekyll s ``` then open http://localhost:4000/ --- ## File: _config.yml # Welcome to Jekyll! # # This config file is meant for settings that affect your whole blog, values # which you are expected to set up once and rarely need to edit after that. # For technical reasons, this file is *NOT* reloaded automatically when you use # 'jekyll serve'. If you change this file, please restart the server process. # Site settings title: DMLC email: muli.cmu@gmail.com DMLC for Scalable and Reliable Machine Learning url: "http://dmlc.ml" # Build settings markdown: redcarpet highlighter: pygments redcarpet: extensions: ["fenced_code_blocks", "no_intra_emphasis", "tables", "autolink", "strikethrough", "with_toc_data"] --- ## File: archive.md --- layout: page title: Archive --- {% for post in site.posts %} * {{ post.date | date_to_string }} » [ {{ post.title }} ]({{ post.url }}) {% endfor %} --- ## File: _posts/2017-01-18-minpy-the-numpy-interface-upon-mxnets-backend.md --- layout: post title: "MinPy: The NumPy Interface upon MXNet’s Backend" date: 2017-01-18 00.00.00 -0800 author: Larry Tang and Minjie Wang comments: true --- Machine learning is now enjoying its golden age. In the past few years, its effectiveness has been proved by solving many traditionally hard problems in computer vision and natural language processing. At the same time, different machine learning frameworks came out to justify different needs. These frameworks, fall generally into two different categories: symbolic programming and imperative programming. ## Symbolic V.S. Imperative Programming Symbolic and imperative programing are two different programming models. Imperative programming are represented by TensorFlow, MXNet’s symbol system, and Theano etc. In symbolic programming model, the execution of a neural network is comprised of two steps. The graph of the computational model needs to be defined first, then the defined graph is sent to execution. For example: ~~~python import numpy as np A = Variable('A') B = Variable('B') C = B * A D = C + Constant(1) # compiles the function f = compile(D) d = f(A=np.ones(10), B=np.ones(10)*2) ~~~ (Example taken from [[1][]]) The core idea here is that the computational graph is defined beforehand, which means the definition and the execution are separated. The advantage of symbolic programming is that it has specified precise computational boundary. Therefore, it is easier to adopt deep optimizations. However, symbolic programming has its limitation. First, it cannot gracefully work with control dependency. Second, it is hard to master a new symbolic language for a newcomer. Third, since the description and the execution of the computational graph are separated, it is difficult to relate execution to value instantiation in the computation. So what about imperative programming? In fact most programmers have already known imperative programming quite well. The everyday C, Pascal, or Python code is almost all imperative. The fundamental idea is that every command is executed step by step, without a separated stage to define the computational graph. For example ~~~python import numpy as np a = np.ones(10) b = np.ones(10) * 2 c = b * a d = c + 1 ~~~ (Example taken from [[1][]]) [1]: http://mxnet.io/architecture/program_model.html When the code executes `c = b * a` and `d = c + 1`, they just run the actual computation. Compared to symbolic programming, imperative programming is much more flexible, since there is no separation of definition and execution. This is important for debugging and visualization. However, its computational boundary is not predefined, leading to harder system optimization. NumPy and Torch adapts imperative programming model. MXNet is a “mixed” framework that aims to provide both symbolic and imperative style, and leaves the choice to its users. While MXNet has a superb symbolic programming subsystem, its imperative subsystem is not powerful enough compared to other imperative frameworks such as NumPy and Torch. This leads to our goal: a fully functional imperative framework that focuses on flexibility without much performance loss, and works well with MXNet’s existing symbol system. ## What is MinPy MinPy aims at providing a high performing and flexible deep learning platform, by prototyping a pure [NumPy](http://www.numpy.org/) interface above [MXNet](https://github.com/dmlc/mxnet) backend. In one word, you get the following automatically with your NumPy code: ~~~python import minpy.numpy as np ~~~ - Operators with GPU support will be ran on GPU. - Graceful fallback for missing operations to NumPy on CPU. - Automatic gradient generation with Autograd support. - Seamless MXNet symbol integration. ## Pure NumPy, purely imperative Why obsessed with NumPy interface? First of all, NumPy is an extension to the Python programming language, with support for large, multi-dimensional arrays, matrices, and a large library of high-level mathematical functions to operate on these abstractions. If you just begin to learn deep learning, you should absolutely start from NumPy to gain a firm grasp of its concepts (see, for example, the Stanford's [CS231n course](http://cs231n.stanford.edu/syllabus.html)). For quick prototyping of advanced deep learning algorithms, you may often start composing with NumPy as well. Second, as an extension of Python, your implementation follows the intuitive imperative style. This is the **only** style, and there is **no** new syntax constructs to learn. To have a taste of this, let's look at some examples below. ## Printing and Debugging In symbolic programming, the control dependency before the print statement is required, otherwise the print operator will not appear on the critical dependency path and thus not being executed. In contrast, MinPy is simply NumPy, as straightforward as Python's hello world. ## Data-dependent branches In symbolic programming, the `lambda` is required in each branch to lazily expand the dataflow graph during runtime, which can be quite confusing. Again, MinPy is NumPy, and you freely use the if statement anyway you like. Tensorflow is just one typical example, many other packages (e.g. Theano, or even MXNet) have similar problems. The underlying reason is the trade-off between *symbolic programming* and *imperative programming*. Code in symbolic programs (Tensorflow and Theano) generates dataflow graph instead of performing concrete computation. This enables extensive optimizations, but requires reinventing almost all language constructs (like if and loop). Imperative programs (NumPy) generates dataflow graph *along with* the computation, enabling you freely query or use the value just computed. In MinPy, we use NumPy syntax to ease your programming, while simultaneously achieving good performance. ## Dynamic automatic gradient computation Automatic gradient computation has become essential in modern deep learning systems. In MinPy, we adopt [Autograd](https://github.com/HIPS/autograd)'s approach to compute gradients. Since the dataflow graph is generated along with the computation, all kinds of native control flow are supported during gradient computation. For example: ~~~python import minpy from minpy.core import grad def foo(x): if x >= 0: return x else: return 2 * x foo_grad = grad(foo) print foo_grad(3) # should print 1.0 print foo_grad(-1) # should print 2.0 ~~~ Here, feel free to use native `if` statement. A complete tutorial about auto-gradient computation can be found [here](https://minpy.readthedocs.io/en/latest/tutorial/autograd_tutorial.html). ## Elegant fallback for missing operators You never like `NotImplementedError`, so do we. NumPy is a very large library. In MinPy, we automatically fallback to NumPy if some operators have not been implemented in MXNet yet. For example, the following code runs smoothly and you don't need to worry about copying arrays back and forth from GPU to CPU; MinPy handles the fallback and its side effect transparently. ~~~python import minpy.numpy as np x = np.zeros((2, 3)) # Use MXNet GPU implementation y = np.ones((2, 3)) # Use MXNet GPU implementation z = np.logaddexp(x, y) # Use NumPy CPU implementation ~~~ ## Seamless MXNet symbol support Although we pick the imperative side, we understand that symbolic programming is necessary for operators like convolution. Therefore, MinPy allows you to "wrap" a symbol into a function that could be called together with other imperative calls. From a programmer's eye, these functions is just as other NumPy calls, thus we preserve the imperative style throughout: ~~~python import mxnet as mx import minpy.numpy as np from minpy.core import Function # Create Function from symbol. net = mx.sym.Variable('x') net = mx.sym.Convolution(net, name='conv', kernel=(3, 3), num_filter=32, no_bias=True) conv = Function(net, input_shapes={'x', (8, 3, 10, 10)} # Call Function as normal function. x = np.zeros((8, 3, 10, 10)) w = np.ones((32, 3, 3, 3,)) y = np.exp(conv(x=x, conv_weight=w)) ~~~ ## Is MinPy fast? The imperative interface does raise many challenges, especially it foregoes some of the deep optimization that only (currently) embodied in symbolic programming. However, MinPy manages to retain reasonable performance, especially when the actual computation is intense. Our next target is to get back the performance with advanced system techniques. ## CS231n -- a Perfect Intro for Newcomers As for deep learning learners, MinPy is a perfect tool to begin with. One of the reasons is that MinPy is fully compatible with NumPy, which means almost no modification to the existing NumPy code. In addition, our team also provides a modified version of CS231n assignments to address the amenity of MinPy. [CS231n](http://cs231n.stanford.edu/) is an introductory course to deep learning taught by Professor [Fei-Fei Li](http://vision.stanford.edu/feifeili/) and her Ph.D students [Andrej Karpathy](http://cs.stanford.edu/people/karpathy/) and [Justin Johnson](http://cs.stanford.edu/people/jcjohns/) at Stanford University. The curriculum covers all the basics of deep learning, including convolutional neural network and recurrent neural network. The course assignments are not only a simple practice of the lecture contents, but a in-depth journey for students to explore deep learning’s latest applications step-by-step. Since MinPy has similar interface with NumPy, our team modified assignments in CS231n to demonstrate the features MinPy provided, in order to create an integrated learning experience for the students who learn deep learning for their first time. The MinPy version of CS231n has already been used in deep learning course at Shanghai Jiao Tong University and ShanghaiTech University. ## Summary MinPy development team began with [Minerva](https://github.com/dmlc/minerva) project. After co-founded MXNet project with the community, we have contributed to the core code of MXNet, including its executor engine, IO, and Caffe operator plugin. Having completed that part, the team decided to take a step back and rethink the user experience, before moving to yet another ambitious stage of performance optimizations. We strive to provide maximum flexibility for users, while creating space for more advanced system optimization. MinPy is pure NumPy and purely imperative. It will be merged into MXNet in the near future. Enjoy! Please send us feedbacks. ## Links * GitHub: * MinPy documentation: ## Acknowledgements - [MXNet Community](http://dmlc.ml/) - Sean Welleck (Ph.D at NYU), Alex Gai and Murphy Li (Undergrads at NYU Shanghai) - Professor [Yi Ma](http://sist.shanghaitech.edu.cn/StaffDetail.asp?id=387) and Postdoc Xu Zhou at ShanghaiTech University; Professor [Kai Yu](https://speechlab.sjtu.edu.cn/~kyu/) and Professor [Weinan Zhang](http://wnzhang.net/) at Shanghai Jiao Tong University. ## MinPy Developers - Project Lead Minjie Wang (NYU) [GitHub](https://github.com/jermainewang) - Larry Tang (U of Michigan*) [GitHub](https://github.com/lryta) - Yutian Li (Stanford) [GitHub](https://github.com/hotpxl) - Haoran Wang (CMU) [GitHub](https://github.com/HrWangChengdu) - Tianjun Xiao (Tesla) [GitHub](https://github.com/sneakerkg) - Ziheng Jiang (Fudan U*) [GitHub](https://github.com/ZihengJiang) - Professor [Zheng Zhang](https://shanghai.nyu.edu/academics/faculty/directory/zheng-zhang) (NYU Shanghai) [GitHub](https://github.com/zzhang-cn) *: MinPy is completed during NYU Shanghai research internship ## Reference 1. --- ## File: _posts/2017-06-01-Generative-Adversarial-Network-in-R.md --- layout: post title: "Conditional Generative Adversarial Network with MXNet R package" date: 2017-06-01 author: Jeremie Desgagne-Bouchard categories: rstats comments: true --- This tutorial shows how to build and train a Conditional Generative Adversarial Network (CGAN) on MNIST images. ### How GAN works A Generative Adversarial Model simultaneously trains two models: a generator that learns to output fake samples from an unknown distribution and a discriminator that learns to distinguish fake from real samples. The CGAN is a conditional variation of the GAN where the generator is instructed to generate a real sample having specific characteristics rather than a generic sample from full distribution. Such condition could be the label associated with an image like in this tutorial or a more detailed tag as shown in the example below: Image credit: [Scott Reed](https://github.com/reedscot/icml2016) ### Initial setup The following packages are needed to run the tutorial: ```r require("imager") require("dplyr") require("readr") require("mxnet") ``` The full demo is comprised of the two following scripts available on [GitHub](https://github.com/dmlc/mxnet/tree/master/example/gan/CGAN_mnist_R): - `CGAN_mnist_setup.R`: prepare data and define the model structure - `CGAN_train.R`: execute the training ### Data preperation The MNIST dataset is available on [Kaggle](https://www.kaggle.com/c/digit-recognizer/data). Once `train.csv` is downloaded into the `data/` folder, we can import into R. ```r train <- read_csv('data/train.csv') train <- data.matrix(train) train_data <- train[,-1] train_data <- t(train_data/255*2-1) train_label <- as.integer(train[,1]) dim(train_data) <- c(28, 28, 1, ncol(train_data)) ``` Custom iterators are defined in `iterators.R` and imported by `CGAN_mnist_setup.R`. ### Generator The generator is a network that creates novel samples (MNIST images) from 2 inputs: - Noise vector - Labels defining the object condition (which digit to produce) The noise vector provides the building blocks to the Generator model, which will learns how to structure that noise into a sample. The `mx.symbol.Deconvolution` operator is used to upsample the initial input from a 1x1 shape up to a 28x28 image. The information on the label for which to generate a fake sample is provided by a one-hot encoding of the label indices that is appended to the random noise. For MNIST, the 0-9 indices are therefore converted into a binary vector of length 10. More complex applications would require embeddings rather than simple one-hot to encode the condition. ### Discriminator The discriminator attempts to distinguish between fake samples produced by the generator and real ones sampled from MNIST training data. In a conditional GAN, the labels associated with the samples are also provided to the Discriminator. In this demo, this information is again provided as a hot-hot encoding of the label that is broadcast to match the image dimensions (10 -> 28x28x10). ### Training logic The training process of the discriminator is most obvious: the loss is simple a binary TRUE/FALSE response and that loss is propagated back into the CNN network. It can therefore be understood as a simple binary classification problem. ```r ### Train loop on fake mx.exec.update.arg.arrays(exec_D, arg.arrays = list(data=D_data_fake, digit=D_digit_fake, label=mx.nd.array(rep(0, batch_size))), match.name=TRUE) mx.exec.forward(exec_D, is.train=T) mx.exec.backward(exec_D) update_args_D<- updater_D(weight = exec_D$ref.arg.arrays, grad = exec_D$ref.grad.arrays) mx.exec.update.arg.arrays(exec_D, update_args_D, skip.null=TRUE) ### Train loop on real mx.exec.update.arg.arrays(exec_D, arg.arrays = list(data=D_data_real, digit=D_digit_real, label=mx.nd.array(rep(1, batch_size))), match.name=TRUE) mx.exec.forward(exec_D, is.train=T) mx.exec.backward(exec_D) update_args_D<- updater_D(weight = exec_D$ref.arg.arrays, grad = exec_D$ref.grad.arrays) mx.exec.update.arg.arrays(exec_D, update_args_D, skip.null=TRUE) ``` The generator loss comes from the backpropagation of the the discriminator loss into its generated output. By faking the generator labels to be real samples into the discriminator, the discriminator back-propagated loss provides the generator with the information on how to best adapt its parameters to trick the discriminator into believing the fake samples are real. This requires to backpropagate the gradients up to the input data of the discriminator (whereas this input gradient is typically ignored in vanilla feedforward network). ```r ### Update Generator weights - use a seperate executor for writing data gradients exec_D_back <- mxnet:::mx.symbol.bind(symbol = D_sym, arg.arrays = exec_D$arg.arrays, aux.arrays = exec_D$aux.arrays, grad.reqs = rep("write", length(exec_D$arg.arrays)), ctx = devices) mx.exec.update.arg.arrays(exec_D_back, arg.arrays = list(data=D_data_fake, digit=D_digit_fake, label=mx.nd.array(rep(1, batch_size))), match.name=TRUE) mx.exec.forward(exec_D_back, is.train=T) mx.exec.backward(exec_D_back) D_grads <- exec_D_back$ref.grad.arrays$data mx.exec.backward(exec_G, out_grads=D_grads) update_args_G <- updater_G(weight = exec_G$ref.arg.arrays, grad = exec_G$ref.grad.arrays) mx.exec.update.arg.arrays(exec_G, update_args_G, skip.null=TRUE) ``` The above training steps are executed in the `CGAN_train.R` script. ### Monitor the training During training, the [imager](http://dahtah.github.io/imager/) package facilitates the visual quality assessment of the fake samples. ```r if (iteration==1 | iteration %% 100==0){ par(mfrow=c(3,3), mar=c(0.1,0.1,0.1,0.1)) for (i in 1:9) { img <- as.array(exec_G$ref.outputs$G_sym_output)[,,,i] plot(as.cimg(img), axes=F) } } ``` Below are samples obtained at different stage of the training. Starting from noise: Slowly getting it - iteration 200: Generate specified digit images on demand - iteration 2400: ### Inference Once the model is trained, synthetic images of the desired digit can be produced by feeding the generator with fixed labels rather than the randomly generated ones used during the training. Here we will generate fake "9": ```r digit <- mx.nd.array(rep(9, times=batch_size)) data <- mx.nd.one.hot(indices = digit, depth = 10) data <- mx.nd.reshape(data = data, shape = c(1,1,-1, batch_size)) exec_G <- mx.simple.bind(symbol = G_sym, data=data_shape_G, ctx = devices, grad.req = "null") mx.exec.update.arg.arrays(exec_G, G_arg_params, match.name=TRUE) mx.exec.update.arg.arrays(exec_G, list(data=data), match.name=TRUE) mx.exec.update.aux.arrays(exec_G, G_aux_params, match.name=TRUE) mx.exec.forward(exec_G, is.train=F) ``` Further details of the CGAN methodology can be found in the paper [Generative Adversarial Text to Image Synthesis](https://arxiv.org/abs/1605.05396). --- ## File: _posts/2017-10-11-rnn-bucket-mxnet-R.md --- layout: post title: "RNN made easy with MXNet R" date: 2017-10-11 author: Jeremie Desgagne-Bouchard categories: rstats comments: true --- This tutorial presents an example of application of RNN to text classification using padded and bucketed data to efficiently handle sequences of varying lengths. Some functionalities require running on a CUDA enabled GPU. Example based on sentiment analysis on the [IMDB data](http://ai.stanford.edu/~amaas/data/sentiment/). What's special about sequence modeling? --------------------------------------- Whether working with times series or text at the character or word level, modeling sequences typically involves dealing with samples of varying length. To efficiently feed the Recurrent Neural Network (RNN) with samples of even length within each batch, two tricks can be used: - Padding: fill the modeled sequences with an arbitrary word/character up to the longest sequence. This results in sequences of even lengths, but potentially of excessive size for an efficient training. - Bucketing: apply the padding trick to subgroups of samples split according to their lengths. It results in multiple training sets, or buckets, within which all samples are padded to an even length. Diagram below illustrates how the two previous samples would be pre-processed if using buckets of size 4 and 6. Non numeric features such as words need to be transformed into a numeric representation. This task is commonly performed by the embedding operator which first requires to convert words into a 0 based index. The embedding will map a vector of features based on that index. In the example below, the embedding projects each word into 2 new numeric features. Data preparation ---------------- For this demo, the data preparation is performed by the script [data_preprocessing_seq_to_one.R](https://github.com/apache/incubator-mxnet/tree/master/example/rnn/bucket_R/data_preprocessing_seq_to_one.R) which involves the following steps: - Import IMDB movie reviews - Split each review into a word vector and apply some common cleansing (remove special characters, lower case, remove extra blank space...) - Convert words into integers and define a dictionary to map the resulting indices with former words - Aggregate the buckets of samples and labels into a list To illustrate the benefit of bucketing, two datasets are created: - `corpus_single_train.rds`: no bucketing, all samples are padded/trimmed to 600 words. - `corpus_bucketed_train.rds`: samples split into 5 buckets of length 100, 150, 250, 400 and 600. Below is the example of the assignation of the bucketed data and labels into `mx.io.bucket.iter` iterator. This iterator behaves essentially the same as the `mx.io.arrayiter` except that is pushes samples coming from the different buckets along with a bucketID to identify the appropriate symbolic graph to use. ``` r corpus_bucketed_train <- readRDS(file = "data/corpus_bucketed_train.rds") corpus_bucketed_test <- readRDS(file = "data/corpus_bucketed_test.rds") vocab <- length(corpus_bucketed_test$dic) ### Create iterators batch.size = 64 train.data.bucket <- mx.io.bucket.iter(buckets = corpus_bucketed_train$buckets, batch.size = batch.size, data.mask.element = 0, shuffle = TRUE) eval.data.bucket <- mx.io.bucket.iter(buckets = corpus_bucketed_test$buckets, batch.size = batch.size, data.mask.element = 0, shuffle = FALSE) ``` Define the architecture ----------------------- Below are the graph representations of a seq-to-one architecture with LSTM cells. Note that input data is of shape `seq.length X batch.size` while the RNN operator requires input of shape `hidden.features X batch.size X seq.length`, requiring to swap axis. For bucketing, a list of symbols is defined, one for each bucket length. During training, at each batch the appropriate symbol is bound according to the bucketID provided by the iterator. ``` r symbol_single <- rnn.graph(config = "seq-to-one", cell_type = "lstm", num_rnn_layer = 1, num_embed = 2, num_hidden = 4, num_decode = 2, input_size = vocab, dropout = 0.5, ignore_label = -1, loss_output = "softmax", output_last_state = F, masking = T) ``` ``` r bucket_list <- unique(c(train.data.bucket$bucket.names, eval.data.bucket$bucket.names)) symbol_buckets <- sapply(bucket_list, function(seq) { rnn.graph(config = "seq-to-one", cell_type = "lstm", num_rnn_layer = 1, num_embed = 2, num_hidden = 4, num_decode = 2, input_size = vocab, dropout = 0.5, ignore_label = -1, loss_output = "softmax", output_last_state = F, masking = T)}) graph.viz(symbol_single, type = "graph", direction = "LR", graph.height.px = 50, graph.width.px = 800, shape=c(64, 5)) ``` The representation of an unrolled RNN typically assumes a fixed length sequence. The operator `mx.symbol.RNN` simplifies the process by abstracting the recurrent cells into a single operator that accepts batches of varying length (each batch contains sequences of identical length). Train the model --------------- First the non bucketed model is trained for 6 epochs: ``` r devices <- mx.gpu(0) initializer <- mx.init.Xavier(rnd_type = "gaussian", factor_type = "avg", magnitude = 2.5) optimizer <- mx.opt.create("rmsprop", learning.rate = 1e-3, gamma1 = 0.95, gamma2 = 0.92, wd = 1e-4, clip_gradient = 5, rescale.grad=1/batch.size) logger <- mx.metric.logger() epoch.end.callback <- mx.callback.log.train.metric(period = 1, logger = logger) batch.end.callback <- mx.callback.log.train.metric(period = 50) system.time( model <- mx.model.buckets(symbol = symbol_single, train.data = train.data.single, eval.data = eval.data.single, num.round = 6, ctx = devices, verbose = FALSE, metric = mx.metric.accuracy, optimizer = optimizer, initializer = initializer, batch.end.callback = NULL, epoch.end.callback = epoch.end.callback) ) ``` ## user system elapsed ## 205.214 17.253 210.265 Then training with the bucketing trick. Note that no additional effort is required: just need to provide a list of symbols rather than a single one and have an iterator pushing samples from the different buckets. ``` r devices <- mx.gpu(0) initializer <- mx.init.Xavier(rnd_type = "gaussian", factor_type = "avg", magnitude = 2.5) optimizer <- mx.opt.create("rmsprop", learning.rate = 1e-3, gamma1 = 0.95, gamma2 = 0.92, wd = 1e-4, clip_gradient = 5, rescale.grad=1/batch.size) logger <- mx.metric.logger() epoch.end.callback <- mx.callback.log.train.metric(period = 1, logger = logger) batch.end.callback <- mx.callback.log.train.metric(period = 50) system.time( model <- mx.model.buckets(symbol = symbol_buckets, train.data = train.data.bucket, eval.data = eval.data.bucket, num.round = 6, ctx = devices, verbose = FALSE, metric = mx.metric.accuracy, optimizer = optimizer, initializer = initializer, batch.end.callback = NULL, epoch.end.callback = epoch.end.callback) ) ``` ## user system elapsed ## 129.578 11.500 125.120 The speedup is substantial, around 125 sec. instead of 210 sec., a 40% speedup with little effort. Plot word embeddings -------------------- Word representation can be visualized by looking at the assigned weights in any of the embedding dimensions. Here, we look simultaneously at the two embeddings learnt in the LSTM model. Since the model attempts to predict the sentiment, it's no surprise that the 2 dimensions into which each word is projected appear correlated with words' polarity. Positive words are associated with lower X1 values ("great", "excellent"), while the most negative words appear at the far right ("terrible", "worst"). By representing words of similar meaning with features of similar values, embedding much facilitates the remaining classification task for the network. Inference on test data ---------------------- The utility function `mx.infer.rnn` has been added to simplify inference on RNN with bucketed data. ``` r ctx <- mx.gpu(0) batch.size <- 64 corpus_bucketed_test <- readRDS(file = "data/corpus_bucketed_test.rds") test.data <- mx.io.bucket.iter(buckets = corpus_bucketed_test$buckets, batch.size = batch.size, data.mask.element = 0, shuffle = FALSE) ``` ``` r infer <- mx.infer.rnn(infer.data = test.data, model = model, ctx = ctx) pred_raw <- t(as.array(infer)) pred <- max.col(pred_raw, tie = "first") - 1 label <- unlist(lapply(corpus_bucketed_test$buckets, function(x) x$label)) acc <- sum(label == pred)/length(label) roc <- roc(predictions = pred_raw[, 2], labels = factor(label)) auc <- auc(roc) ``` Accuracy: 87.6% AUC: 0.9436