The 14 Best Data Visualization Tools

http://thenextweb.com/dd/2015/04/21/the-14-best-data-visualization-tools/

1. D3.js, short for ‘Data Driven Documents (open source)
2. FusionCharts (commercial)
3. Chart.js (open source)
4. Google Charts
5. Highcharts  (free for non commercial)
6. Leaflet  (open source)
7. dygraphs (open source)
8. Datawrapper  (online)
9. Tableau Public (free)

Rodeo

https://www.yhat.com/products/rodeo/downloads

https://github.com/zer0n/deepframeworks/

Evaluation of Deep Learning Toolkits

Abstract. In this study, I evaluate some popular deep learning toolkits. The candidates are listed in alphabetical order: Caffe, CNTK, TensorFlow, Theano, and Torch. This is a dynamic document and the evaluation, to the best of my knowledge, is based on the current state of their code.
I also provide ratings in some areas because for a lot of people, ratings are useful. However, keep in mind that ratings are inherently subjective [1].
If you find something wrong or inadequate, please help improve by filing an issue.
Table of contents

1. Modeling Capability
2. Interfaces
3. Model Deployment
4. Performance
5. Architecture
6. Ecosystem
7. Cross-platform

---

Modeling Capability

In this section, we evaluate each toolkit's ability to train common and state-of-the-art networks without writing too much code. Some of these networks are:

• ConvNets: AlexNet, OxfordNet, GoogleNet
• RecurrentNets: plain RNN, LSTM/GRU, bidirectional RNN
• Sequential modeling with attention.

In addition, we also evaluate the flexibility to create a new type of model.

Caffe

Caffe is perhaps the first mainstream industry-grade deep learning toolkit, started in late 2013, due to its excellent convnet implementation (at the time). It is still the most popular toolkit within the computer vision community, with many extensions being actively added.
However, its support for recurrent networks and language modeling in general is poor, due to its legacy architecture, which's limitations are detailed in the architecture section.

CNTK

CNTK is a deep learning system started by the speech people who started the deep learning craze and grown into a more general platform-independent deep learning system. It is better known in the speech community than in the general deep learning community.
In CNTK (as in TensorFlow and Theano), a network is specified as a symbolic graph of vector operations, such as matrix add/multiply or convolution. A layer is just a composition of those operations. The fine granularity of the building blocks (operations) allows users to invent new complex layer types without implementing them in a low-level language (as in Caffe).
More to be written ...

TensorFlow

State-of-the-art models

• RNN API and implementation are suboptimal. The team also commented about it here and here.
• Bidirectional RNN not available yet
• No 3D convolution, which is useful for video recognition

New models Since TF uses symbolic graph of vector operations approach, specifying a new network is fairly easy.
The public release of TF doesn’t yet support loop and condition controls in the graph definition. This makes RNN implementations less ideal because they have to use Python loops and no graph compiler optimization can be made.
Google claimed to have this in their white paper and details are still being worked out.

Theano

State-of-the-art models. Theano has implementation for most state-of-the-art networks, either in the form of a higher-level framework (e.g. Blocks, Keras, etc.) or in pure Theano. In fact, many recent research ideas (e.g. attentional model) started here.
New models. Theano pioneered the trend of using symbolic graph for programming a network. Theano's symbolic API supports looping control, so-called scan, which makes implementing RNNs easy and efficient. Users don't always have to define a new model at the tensor operations level. There are a few higher-level frameworks, mentioned above, which make model definition and training simpler.

Torch

State-of-the-art models

• Excellent for conv nets. It's worth noting that temporal convolution can be done in TensorFlow/Theano via conv2d but that's a trick. The native interface for temporal convolution in Torch makes it slightly more intuitive to use.
• Rich set of RNNs available through a non-official extension [2]

New models. In Torch, there are multiple ways (stack of layers or graph of layers) to define a network but essentially, a network is defined as a graph of layers. Because of this coarser granularity, Torch is sometimes considered less flexible because for new layer types, users have to implement the full forward, backward, and gradient input update.
However, unlike Caffe, defining a new layer in Torch is much easier because you don't have to program in C++. Plus, in Torch, the difference between new layer definition and network definition is minimal. In Caffe, layers are defined in C++ while networks are defined via Protobuf.
Hence, in my opinion, Torch is as flexible as Theano.

---

Left: graph model of CNTK/Theano/TensorFlow; Right: graph model of Caffe/Torch

Interfaces

Caffe

Caffe has pycaffe interface but that's a mere secondary alternative to the command line interface. The model has to be defined in protobuf (usually with a plain text editor), even if you use pycaffe.

CNTK

The way to use CNTK, similar to Caffe, is to specify a config file and run command line. CNTK is slightly worse than Caffe because there's no Python or any other high-level language interface.

TensorFlow

TF supports two interfaces: Python and C++. This means that you can do experiments in a rich, high-level environment and deploy your model in an environment that requires native code or low latency.
It would be perfect if TF supports F# or TypeScript. The lack of static type in Python is just ... painful :).

Theano

Python

Torch

Torch runs on LuaJIT, which is amazingly fast (comparable with industrial languages such as C++/C#/Java). Hence developers don't have to think about symbolic programming, which can be limited. They can just write all kinds of computations without worrying about performance penalty.
However, let's face it, Lua is not yet a mainstream language.

Model Deployment

How easy to deploy a new model?

Caffe

Caffe is C++ based, which can be compiled on a variety of devices. It is cross-platform (windows port is available and maintained here). Which makes Caffe the best choice with respect deployment.

CNTK

Like Caffe, CNTK is also C++ based and is cross-platform. Hence, deployment should be easy in most cases. However, to my understanding, it doesn't work on ARM architecture, which limits its its capability on mobile devices.

TensorFlow

TF supports C++ interface and the library can be compiled/optimized on ARM architectures because it uses Eigen (instead of a BLAS library). This means that you can deploy your trained models on a variety of devices (servers or mobile devices) without having to implement a separate model decoder or load Python/LuaJIT interpreter [3].
TF doesn't work on Windows yet so TF models can't be deployed on Windows devices though.

Theano

The lack of low-level interface and the inefficiency of Python interpreter makes Theano less attractive for industrial users. For a large model, the overhead of Python isn’t too bad but the dogma is still there.
The cross-platform nature (mentioned below) enables a Theano model to be deployed in a Windows environment. Which helps it gain some points.

Torch

Torch require LuaJIT to run models. This makes it less attractive than bare bone C++ support of Caffe/CNTK/TF. It’s not just the performance overhead, which is minimal. The bigger problem is integration, at API level, with a larger production pipeline.

Performance

Single-GPU

All of these toolkits call cuDNN so as long as there’s no major computations or memory allocations at the outer level, they should perform similarly.
Soumith@FB has done some benchmarking for ConvNets. Deep Learning is not just about feedforward convnets, not just about ImageNet, and certainly not just about a few passes over the network. However, Soumith’s benchmark is the only notable one as of today. So we will base the Single-GPU performance rating based on his benchmark.

Caffe

For what it can do, Caffe is simply fast.

CNTK

For what it can do, CNTK is simply fast.

TensorFlow

TF only uses cuDNN v2 and even so, its performance is ~1.5x slower than Torch with cuDNN v2. It also runs out of memory when training GoogleNet with batch size 128. More details here.
A few issues have been identified in that thread: excessive memory allocation, different tensor layout from cuDNN’s, no in-place op, etc.

Theano

On big networks, Theano’s performance is on par with Torch7, according to this benchmark. The main issue of Theano is startup time, which is terrible, because Theano has to compile C/CUDA code to binary. We don’t always train big models. In fact, DL researchers often spend more time debugging than training big models. TensorFlow doesn’t have this problem. It simply maps the symbolic tensor operations to the already-compiled corresponding function calls.
Even import theano takes time because this import apparently does a lot of stuffs. Also, after import Theano, you are stuck with a pre-configured device (e.g. GPU0).

Torch

Simply awesome without the *bugs* that TensorFlow and Theano have.

Multi-GPU

According to a recent benchmarking study by the CNTK team (see figure below), CNTK simply leads the pack in the multi-GPU and distributed deep learning regime. CNTK implements 1-bit SGD and adaptive minibatching in Seide et al., which are truly gems for distributed training. This capability is unique among all deep learning toolkits.

Source: https://cntk.codeplex.com Disclaimer: per CNTK team's request, I helped providing Torch and Theano (via Keras) measurements.

Architecture

Developer Zone

Caffe

Caffe's architecture was considered excellent when it was born but in the modern standard, it is considered average. The main pain points of Caffe are its layer-wise design in C++ and the protobuf interface for model definition.
Layer-wise design. The building block of a network in Caffe is layer.

• For new layer types, you have to define the full forward, backward, and gradient update. You can see an already long-list of layers implemented in (official) caffe.
• What's worse is that if you want to support both CPU and GPU, you need to implement extra functions, e.g. Forward_gpu and Backward_gpu.
• Worse, you need to assign an int id to your layer type and add that to the proto file. If your pull request is not merged early, you may need to change the id because someone else already claims that.

Protobuf. Caffe has pycaffe interface but that's a mere replacement of the command line interface. The model has to be defined in protobuf (usually with a plain text editor), even if you use pycaffe.
[Copied from my own answer on Quora]

CNTK

To be updated ...

TensorFlow

TF has a clean, modular architecture with multiple frontends and execution platforms. Details are in the white paper.

Theano

The architecture is fairly hacky: the whole code base is Python where C/CUDA code is packaged as Python string. This makes it hard to navigate, debug, refactor, and hence contribute as developers.

Torch

Torch7 and nn libraries are also well-designed with clean, modular interfaces.

Ecosystem

• Caffe and CNTK: C++
• TensorFlow: Python and C++
• Theano: Python
• Torch: Lua is not a mainstream language and hence libraries built for it are not as rich as ones built for Python.

Cross-platform

Caffe, CNTK, and Theano work on all OSes. TensorFlow and Torch do not work on Windows and there's no known plan to port from either camp.

---

Footnotes
[1] Note that I don’t aggregate ratings because different users/developers have different priorities.
[2] Disclaimer: I haven’t analyzed this extension carefully.
[3] See my blog post for why this is desirable.

Deep Learning An MIT Press book in preparation Ian Goodfellow, Aaron Courville, and Yoshua Bengio

http://goodfeli.github.io/dlbook/

FAQ

• Can I get a PDF of this book? No, our contract with MIT Press forbids distribution of too easily copied electronic formats of the book. Google employees who would like a paper copy of the book can send Ian the name of the printer nearest their desk and he will send a print job to that printer containing as much of the book as you would like to read.
  
• Why are you using HTML format for the drafts? This format is a sort of weak DRM. It's intended to discourage unauthorized copying/editing of the book. Unfortunately, the conversion from PDF to HTML is not perfect, and some things like subscript expressions do not render correctly. If you have a suggestion for a better way of making the book available to a wide audience while preventing unauthorized copies, please let us know.
• What is the best way to print the HTML format?
Printing seems to work best printing directly from the browser, using Chrome.

• When will the book come out? It's difficult to predict. We're doing our best to finish it as soon as possible. We expect to deliver the manuscript to reviews by January, but there will be some delay with review, typesetting, etc. Please contact us if you are interested in using the textbook for course materials in the short term, we may be able to find a way to accommodate you.
  

Draft Version 2015-12-17

Please help us make this a great book! This is still only a draft and can be improved in many ways. We've included some of our TODO marks to help you see which sections are still under rapid development. If you see a section that we appear to have finished writing and you find it unclear or inaccurate, please let us know. Please specify (by date) which version you are commenting on. Do not hesitate to contact any of the authors directly by e-mail: Ian (<lastname>@google.com), Aaron (<firstname>.<lastname>@gmail.com), Yoshua (<firstname>.<lastname>@umontreal.ca).

• Table of Contents
• Acknowledgements
• Notation
• 1 Introduction
• Part I: Applied Math and Machine Learning Basics
• 2 Linear Algebra
• 3 Probability and Information Theory
• 4 Numerical Computation
• 5 Machine Learning Basics
• Part II: Modern Practical Deep Networks
• 6 Feedforward Deep Networks
• 7 Regularization
• 8 Optimization for Training Deep Models
• 9 Convolutional Networks
• 10 Sequence Modeling: Recurrent and Recursive Nets
• 11 Practical Methodology
• 12 Applications
• Part III: Deep Learning Research
• 13 Linear Factor Models
• 14 Autoencoders
• 15 Representation Learning
• 16 The Manifold Perspective on Representation Learning
• 17 Structured Probabilistic Models for Deep Learning
• 18 Monte Carlo Methods
• 19 Confronting the Partition Function
• 20 Approximate Inference
• 21 Deep Generative Models
• Bibliography
• Index

Creatures Avoiding Planks

http://blog.otoro.net/2015/05/07/creatures-avoiding-planks/

Researchers outside Google are testing the software that the company uses to add artificial intelligence to many of its products.

http://www.technologyreview.com/news/544356/heres-what-developers-are-doing-with-googles-ai-brain/

An artificial intelligence engine that Google uses in many of its products, and that it made freely available last month, is now being used by others to perform some neat tricks, including translating English into Chinese, reading handwritten text, and even generating original artwork.
The AI software, called TensorFlow, provides a straightforward way for users to train computers to perform tasks by feeding them large amounts of data. The software incorporates various methods for efficiently building and training simulated “deep learning” neural networks across different computer hardware.
Deep learning is an extremely effective technique for training computers to recognize patterns in images or audio, enabling machines to perform with human-like competence useful tasks such as recognizing faces or objects in images. Recently, deep learning also has shown significant promise for parsing natural language, by enabling machines to respond to spoken or written queries in meaningful ways.
Speaking at the Neural Information Processing Society (NIPS) conference in Montreal this week, Jeff Dean, the computer scientist at Google who leads the TensorFlow effort, said that the software is being used for a growing number of experimental projects outside the company.
These include software that generates captions for images and code that translates the documentation for TensorFlow into Chinese. Another project uses TensorFlow to generate artificial artwork. “It’s still pretty early,” Dean said after the talk. “People are trying to understand what it’s best at.”
TensorFlow grew out of a project at Google, called Google Brain, aimed at applying various kinds of neural network machine learning to products and services across the company. The reach of Google Brain has grown dramatically in recent years. Dean said that the number of projects at Google that involve Google Brain has grown from a handful in early 2014 to more than 600 today.
Most recently, the Google Brain helped develop Smart Reply, a system that automatically recommends a quick response to messages in Gmail after it scans the text of an incoming message. The neural network technique used to develop Smart Reply was presented by Google researchers at the NIPS conference last year.
Dean expects deep learning and machine learning to have a similar impact on many other companies. “There is a vast array of ways in which machine learning is influencing lots of different products and industries,” he said. For example, the technique is being tested in many industries that try to make predictions from large amounts of data, ranging from retail to insurance.

Google was able to give away the code for TensorFlow because the data it owns is a far more valuable asset for building a powerful AI engine. The company hopes that the open-source code will help it establish itself as a leader in machine learning and foster relationships with collaborators and future employees. TensorFlow “gives us a common language to speak, in some sense,” Dean said. “We get benefits from having people we hire who have been using TensorFlow. It’s not like it’s completely altruistic.”
A neural network consists of layers of virtual neurons that fire in a cascade in response to input. A network “learns” as the sensitivity of these neurons is tuned to match particular input and output, and having many layers makes it possible to recognize more abstract features, such as a face in a photograph.
TensorFlow is now one of several open-source deep learning software libraries, and its performance currently lags behind some other libraries for certain tasks. However, it is designed to be easy to use, and it can easily be ported between different hardware. And Dean says his team is hard at work trying to improve its performance.
In the race to dominate machine learning and attract the best talent, however, other companies may release competing AI engines of their own.