Lessons from My First Two Years of AI Research

http://web.mit.edu/tslvr/www/lessons_two_years.html

Artificial Intelligence — The Revolution Hasn’t Happened Yet

https://medium.com/@mijordan3/artificial-intelligence-the-revolution-hasnt-happened-yet-5e1d5812e1e7

Artificial Intelligence (AI) is the mantra of the current era. The phrase is intoned by technologists, academicians, journalists and venture capitalists alike. As with many phrases that cross over from technical academic fields into general circulation, there is significant misunderstanding accompanying the use of the phrase. But this is not the classical case of the public not understanding the scientists — here the scientists are often as befuddled as the public. The idea that our era is somehow seeing the emergence of an intelligence in silicon that rivals our own entertains all of us — enthralling us and frightening us in equal measure. And, unfortunately, it distracts us.

The US Military Needs to Urgently Rethink its Deep Learning Strategy

https://medium.com/intuitionmachine/the-us-military-is-dangerously-behind-with-deep-learning-2d929929e595

Here’s how the US needs to prepare for the age of artificial intelligence

https://www.technologyreview.com/s/610379/heres-how-the-us-needs-to-prepare-for-the-age-of-artificial-intelligence/

Here’s how the US needs to prepare for the age of artificial intelligence
Government indifference toward AI could let the US lose ground to rival countries. But what would a good AI plan actually look like?


Politicians worldwide are stealing one of the US government’s best ideas by drawing up ambitious plans to make the most of advances in artificial intelligence.
These AI manifestos, penned in Paris, Beijing, and elsewhere, follow the example of the Obama administration, which released a report on the technology toward the end of its tenure. This report did not include funding, but it made it clear that AI should be a key focus of government strategy.
The Trump administration has abandoned this vision and has no intention of devising its own AI plan, say those working there. They say there is no need for an AI moonshot, and that minimizing government interference is the best way to make sure the technology flourishes.
That looks like a huge mistake. If it essentially ignores such a technological transformation, the US might never make the most of an opportunity to reboot its economy and kick-start both wage growth and job creation. Failure to plan could also cause the birthplace of AI to lose ground to international rivals.
France is the latest country to bet heavily on AI. Last week, President Emmanuel Macron announced an AI plan that includes $1.6 billion in funding, new research centers, data-sharing initiatives, and ethical guidelines. Macron also echoed a view held by many economists and policy experts by suggesting that the technology could boost overall economic productivity and lead to wage growth and job creation.

Sign up for the The Algorithm

News and views on the latest in artificial intelligence

Manage your newsletter preferences

The most notable national AI plan has come from China, which last year announced a plan to dominate the industry by 2030 (see “China’s AI awakening”). Details of that undertaking are murky, but many billions of dollars will be fed into the industry. And experts there are also debating how the technology might affect the economy and society.
Other countries have also recognized the technology’s potential. The UK government has published several major studies on AI, and last year it committed $663 million to tech funding, much of which has been earmarked for AI. Canada has created several new AI institutes and introduced incentives to attract companies working on the technology. Germany is said to be preparing its own AI strategy.
If the US were to draw up a new AI master plan, what should it aim to do?

Invest more

Funding AI research is the biggest priority, says Jason Furman, one of Obama’s top economic advisors and author of the 2016 AI report. Furman says basic research requires government backing and that advances made privately won’t benefit the country as a whole.
Furman adds that it is especially important to promote AI because it could provide precisely the economic boost that leads to wage growth and new employment opportunities. “Economists have found, in general, that we may be spending about a quarter of what we should be, based on the returns,” he says.
Oren Etzioni, CEO of the Allen Institute for Artificial Intelligence, a nonprofit based in Seattle, points out that core AI breakthroughs have their origins in academia. “If you don’t fund the universities, you run the risk of starving the goose that lays the golden egg,” he says. “We are at a momentous point in history.”

Recommended for You

1. In blockchain we trust
2. Mark Zuckerberg says fixing Facebook’s data scandal will “significantly impact” its profitability
3. Even if Magic Leap’s goggles suck, it can probably still make money off them
4. DNA tests for IQ are coming, but it might not be smart to take one
5. Facebook’s disappearing message saga is the act of a company in turmoil

Advisors to the White House reject the idea that the government isn’t spending enough. They point out that billions of dollars are funneled into projects related to AI through the Department of Defense, the National Science Foundation, and the Department of Energy. The omnibus spending bill approved by Congress will also provide new funds for academic research.

Prepare for job losses

But a government AI policy must go beyond calls for research funding.
While AI can drive economic growth, it may also accelerate the eradication of some occupations, transform the nature of work in other jobs, and exacerbate economic inequality (see “The relentless pace of automation”). It is critical that governments prepare for this transformation. This might mean exploring ways to find training and employment opportunities for those who have lost jobs to automation and AI. Academic experts and institutions have been sounding the alarm over this issue and have warned that it might have serious social consequences. Those problems will surely require government action.
“Planning for job displacement that AI will cause is best done by government,” says Andrew Ng, a prominent AI researcher who was previously the chief scientist at Baidu and is now involved with several different AI projects.

Nurture talent

Another key consideration should be attracting and holding onto talented AI experts. The US has a long history of drawing the top academic talent, and this has fed directly into industry. The academics most associated with the field of deep learning, for example—Geoffrey Hinton, Yann LeCun, Yoshua Bengio, and Andrew Ng—were all born in other countries but came to universities and companies in the US.

Related Story

China’s AI Awakening
中国 人工智能 的崛起

The West shouldn’t fear China’s artificial-intelligence revolution. It should copy it.

Other countries now aim to retain or recapture talent. The UK government commissioned an AI report not long after Google acquired DeepMind, a UK startup that has made fundamental AI advances, in 2014. “[The government] said we want to make sure the UK is well positioned to have jobs in AI, and we want to keep our startups in the UK,” says Dame Wendy Hall, a professor at Southampton University and the author of the report. It recommends, for instance, measures that would help startups access more data and funding, perhaps making them less likely to agree to an acquisition.
The US government is currently on the opposite path. By tightening its visa program, taking an aggressive stance on immigration, and threatening funding for graduate students, it is turning talent away (see “The US leads in artificial intelligence, but for how long?”). “The openness of our society is a huge advantage,” Etzioni adds. “If we lose our openness on immigration, we are putting a nail in our scientific coffin.”

Prioritize education

Education should be a key part of the picture. New AI scientists will fuel the industry, but broader AI expertise across different industries is also an important goal.
Tess Posner, who leads the nonprofit AI4All, says beyond boosting the industry, education can help address the its shortcomings. “There is not only a talent crisis in AI; there is a diversity crisis, and that is problematic,” she says.
Posner suggests that the government should not be responsible for educating AI experts, but it should be at the forefront of the effort. “It’s such an important tech that there should be a national focus on it,” she says.
Again, other countries are stealing a march here. The UK has, for example, launched new university courses focused on AI and added funding for doctoral students at top universities.

Guide regulation

The US doesn’t necessarily need new rules and regulations for AI, but there may be situations where the government could step in to help apply existing laws. In the case of autonomous cars, for example, adapting existing regulations had been an effective approach. Recent accidents involving self-driving vehicles are being investigated by federal agencies to make sure no rules were broken.
The same is true in in dealing with algorithmic bias, says Solon Barocas, an assistant professor at Cornell University and an expert on fairness in machine learning. New York City has launched an effort to make algorithmic decision-making more accountable using the existing regulatory framework.
But Barocas says regulators may need guidance in figuring out how to apply current laws. “There are limits to how far you can go with existing regulations,” he says. “There needs to be a lot more work done to help regulators.”
It may also be important to consider new challenges that might arise as a result of AI. A report authored recently by academics and industry experts speculates about ways that AI could be misused by criminals, terrorists, or foreign governments.

Understand the technology

Above all, the government needs to understand what AI is and what it will do. Since artificial intelligence is such a complex and fast-moving field, it is especially important for experts to be brought in to brief policymakers and the administration. Without technical acumen, it will be a challenge to act effectively in any area relevant to AI.
But technical expertise is distressingly lacking in the US government today. There is no leader at the Office of Science and Technology Policy, a key role in advising the president, and many other jobs there reportedly remain unfilled.
“Whenever there is technological disruption, leadership matters,” says Ng.

Machine Learning Workflow Links

Machine Learning for Modelling Real World Dynamical System shttps://www.cc.gatech.edu/~lsong/teaching/CSE6740fall14/BBoots.pdf

Machine Learning Workflow on Diabetes Data : Part 01 https://towardsdatascience.com/machine-learning-workflow-on-diabetes-data-part-01-573864fcc6b8

Machine Learning Workflow on Diabetes Data : Part 02 https://towardsdatascience.com/machine-learning-workflow-on-diabetes-data-part-02-11262b7f7a5c

Machine Learning Workflow http://fgiasson.com/blog/index.php/2017/03/10/a-machine-learning-workflow/

Understanding the Machine Learning Workflow https://www.packtpub.com/mapt/book/big_data_and_business_intelligence/9781783980284/2/ch02lvl1sec14/understanding-the-machine-learning-workflow

Machine Learning Workflow https://deeplearning4j.org/machine-learning-workflow

Machine Learning Workflows https://www.kdnuggets.com/2017/05/machine-learning-workflows-python-scratch-part-1.html

Machine Learning Pipeline https://sigmoidal.io/machine-learning-pipeline/

Machine Learning Workflow https://cloud.google.com/ml-engine/docs/tensorflow/ml-solutions-overview

A Brief History of Deep Learning

http://www.dataversity.net/brief-history-deep-learning/


Deep Learning, as a branch of Machine Learning, employs algorithms to process data and imitate the thinking process, or to develop abstractions. Deep Learning (DL) uses layers of algorithms to process data, understand human speech, and visually recognize objects. Information is passed through each layer, with the output of the previous layer providing input for the next layer. The first layer in a network is called the input layer, while the last is called an output layer. All the layers between the two are referred to as hidden layers. Each layer is typically a simple, uniform algorithm containing one kind of activation function.
Feature extraction is another aspect of Deep Learning. Feature extraction uses an algorithm to automatically construct meaningful “features” of the data for purposes of training, learning, and understanding. Normally the Data Scientist, or programmer, is responsible for feature extraction.
The history of Deep Learning can be traced back to 1943, when Walter Pitts and Warren McCulloch created a computer model based on the neural networks of the human brain. They used a combination of algorithms and mathematics they called “threshold logic” to mimic the thought process. Since that time, Deep Learning has evolved steadily, with only two significant breaks in its development. Both were tied to the infamous Artificial Intelligence winters.
Henry J. Kelley is given credit for developing the basics of a continuous Back Propagation Model in 1960. In 1962, a simpler version based only on the chain rule was developed by Stuart Dreyfus. While the concept of back propagation (the backward propagation of errors for purposes of training) did exist in the early 1960s, it was clumsy and inefficient, and would not become useful until 1985.
The earliest efforts in developing Deep Learning algorithms came from Alexey Grigoryevich Ivakhnenko (developed the Group Method of Data Handling) and Valentin Grigorʹevich Lapa (author of Cybernetics and Forecasting Techniques) in 1965. They used models with polynomial (complicated equations) activation functions, that were then analyzed statistically. From each layer, the best statistically chosen features were then forwarded on to the next layer (a slow, manual process).
During the 1970’s the first AI winter kicked in, the result of promises that couldn’t be kept. The impact of this lack of funding limited both DL and AI research. Fortunately, there were individuals who carried on the research without funding.
The first “convolutional neural networks” were used by Kunihiko Fukushima. Fukushima designed neural networks with multiple pooling and convolutional layers. In 1979, he developed an artificial neural network, called Neocognitron, which used a hierarchical, multilayered design. This design allowed the computer the “learn” to recognize visual patterns. The networks resembled modern versions, but were trained with a reinforcement strategy of recurring activation in multiple layers, which gained strength over time. Additionally, Fukushima’s design allowed important features to be adjusted manually by increasing the “weight” of certain connections.
Many of the concepts of Neocognitron continue to be used. The use of top-down connections and new learning methods have allowed for a variety of neural networks to be realized. When more than one pattern is presented at the same time, the Selective Attention Model can separate and recognize individual patterns by shifting its attention from one to the other. (The same process many of us use when multitasking). A modern Neocognitron can not only identify patterns with missing information (for example, an incomplete number 5), but can also complete the image by adding the missing information. This could be described as “inference.”
Back propagation, the use of errors in training Deep Learning models, evolved significantly in 1970. This was when Seppo Linnainmaa wrote his master’s thesis, including a FORTRAN code for back propagation. Unfortunately, the concept was not applied to neural networks until 1985. This was when Rumelhart, Williams, and Hinton demonstrated back propagation in a neural network could provide “interesting” distribution representations. Philosophically, this discovery brought to light the question within cognitive psychology of whether human understanding relies on symbolic logic (computationalism) or distributed representations (connectionism). In 1989, Yann LeCun provided the first practical demonstration of backpropagation at Bell Labs. He combined convolutional neural networks with back propagation onto read “handwritten” digits. This system was eventually used to read the numbers of handwritten checks.
This time is also when the second AI winter (1985-90s) kicked in, which also effected research for neural networks and Deep Learning. Various overly-optimistic individuals had exaggerated the “immediate” potential of Artificial Intelligence, breaking expectations and angering investors. The anger was so intense, the phrase Artificial Intelligence reached pseudoscience status. Fortunately, some people continued to work on AI and DL, and some significant advances were made. In 1995, Dana Cortes and Vladimir Vapnik developed the support vector machine (a system for mapping and recognizing similar data). LSTM (long short-term memory) for recurrent neural networks was developed in 1997, by Sepp Hochreiter and Juergen Schmidhuber.
The next significant evolutionary step for Deep Learning took place in 1999, when computers started becoming faster at processing data and GPU (graphics processing units) were developed. Faster processing, with GPUs processing pictures, increased computational speeds by 1000 times over a 10 year span. During this time, neural networks began to compete with support vector machines. While a neural network could be slow compared to a support vector machine, neural networks offered better results using the same data. Neural networks also have the advantage of continuing to improve as more training data is added.
Around the year 2000, The Vanishing Gradient Problem appeared. It was discovered “features” (lessons) formed in lower layers were not being learned by the upper layers, because no learning signal reached these layers. This was not a fundamental problem for all neural networks, just the ones with gradient-based learning methods. The source of the problem turned out to be certain activation functions. A number of activation functions condensed their input, in turn reducing the output range in a somewhat chaotic fashion. This produced large areas of input mapped over an extremely small range. In these areas of input, a large change will be reduced to a small change in the output, resulting in a vanishing gradient. Two solutions used to solve this problem were layer-by-layer pre-training and the development of long short-term memory.
In 2001, a research report by META Group (now called Gartner) described he challenges and opportunities of data growth as three-dimensional. The report described the increasing volume of data and the increasing speed of data as increasing the range of data sources and types. This was a call to prepare for the onslaught of Big Data, which was just starting.
In 2009, Fei-Fei Li, an AI professor at Stanford launched ImageNet, assembled a free database of more than 14 million labeled images. The Internet is, and was, full of unlabeled images. Labeled images were needed to “train” neural nets. Professor Li said, “Our vision was that Big Data would change the way machine learning works. Data drives learning.”
By 2011, the speed of GPUs had increased significantly, making it possible to train convolutional neural networks “without” the layer-by-layer pre-training. With the increased computing speed, it became obvious Deep Learning had significant advantages in terms of efficiency and speed. One example is AlexNet, a convolutional neural network whose architecture won several international competitions during 2011 and 2012. Rectified linear units were used to enhance the speed and dropout.
Also in 2012, Google Brain released the results of an unusual project known as The Cat Experiment. The free-spirited project explored the difficulties of “unsupervised learning.” Deep Learning uses “supervised learning,” meaning the convolutional neural net is trained using labeled data (think images from ImageNet). Using unsupervised learning, a convolutional neural net is given unlabeled data, and is then asked to seek out recurring patterns.
The Cat Experiment used a neural net spread over 1,000 computers. Ten million “unlabeled” images were taken randomly from YouTube, shown to the system, and then the training software was allowed to run. At the end of the training, one neuron in the highest layer was found to respond strongly to the images of cats. Andrew Ng, the project’s founder said, “We also found a neuron that responded very strongly to human faces.” Unsupervised learning remains a significant goal in the field of Deep Learning.
The Cat Experiment works about 70% better than its forerunners in processing unlabeled images. However, it recognized less than a 16% of the objects used for training, and did even worse with objects that were rotated or moved.
Currently, the processing of Big Data and the evolution of Artificial Intelligence are both dependent on Deep Learning. Deep Learning is still evolving and in need of creative ideas.

Companion webpage to the book "Mathematics for Machine Learning"

https://mml-book.github.io/

CS224n: Natural Language Processing with Deep Learning

http://web.stanford.edu/class/cs224n/reports.html

Visualizing Representations: Deep Learning and Human Beings

http://colah.github.io/posts/2015-01-Visualizing-Representations/

Posted on January 16, 2015

data visualization, machine learning, word embeddings, neural networks, deep learning, user interface, wikipedia

In a previous post, we explored techniques for visualizing high-dimensional data. Trying to visualize high dimensional data is, by itself, very interesting, but my real goal is something else. I think these techniques form a set of basic building blocks to try and understand machine learning, and specifically to understand the internal operations of deep neural networks.

Deep neural networks are an approach to machine learning that has revolutionized computer vision and speech recognition in the last few years, blowing the previous state of the art results out of the water. They’ve also brought promising results to many other areas, including language understanding and machine translation. Despite this, it remains challenging to understand what, exactly, these networks are doing.

I think that dimensionality reduction, thoughtfully applied, can give us a lot of traction on understanding neural networks.

Understanding neural networks is just scratching the surface, however, because understanding the network is fundamentally tied to understanding the data it operates on. The combination of neural networks and dimensionality reduction turns out to be a very interesting tool for visualizing high-dimensional data – a much more powerful tool than dimensionality reduction on its own.

As we dig into this, we’ll observe what I believe to be an important connection between neural networks, visualization, and user interface.

The $1700 great Deep Learning box: Assembly, setup and benchmarks

https://blog.slavv.com/the-1700-great-deep-learning-box-assembly-setup-and-benchmarks-148c5ebe6415

Building a desktop after a decade of MacBook Airs and cloud servers

After years of using a thin client in the form of increasingly thinner MacBooks, I had gotten used to it. So when I got into Deep Learning (DL), I went straight for the brand new at the time Amazon P2 cloud servers. No upfront cost, the ability to train many models simultaneously and the general coolness of having a machine learning model out there slowly teaching itself.

However, as time passed, the AWS bills steadily grew larger, even as I switched to 10x cheaper Spot instances. Also, I didn’t find myself training more than one model at a time. Instead, I’d go to lunch/workout/etc. while the model was training, and come back later with a clear head to check on it.

The struggle is real

But eventually the model complexity grew and took longer to train. I’d often forget what I did differently on the model that had just completed its 2-day training. Nudged by the great experiences of the other folks on the Fast.AI Forum, I decided to settle down and to get a dedicated DL box at home.

The most important reason was saving time while prototyping models — if they trained faster, the feedback time would be shorter. Thus it would be easier for my brain to connect the dots between the assumptions I had for the model and its results.

Then I wanted to save money — I was using Amazon Web Services (AWS), which offered P2 instances with Nvidia K80 GPUs. Lately, the AWS bills were around $60–70/month with a tendency to get larger. Also, it is expensive to store large datasets, like ImageNet.

And lastly, I haven’t had a desktop for over 10 years and wanted to see what has changed in the meantime (spoiler alert: mostly nothing).

What follows are my choices, inner monologue, and gotchas: from choosing the components to benchmarking.

Table of contents

1. Choosing components

2. Putting it together

3. Software Setup

4. Benchmarks

---

Choosing the components

A sensible budget for me would be about 2 years worth of my current compute spending. At $70/month for AWS, this put it at around $1700 for the whole thing.

You can check out all the components used. The PC Part Picker site is also really helpful in detecting if some of the components don’t play well together.

GPU

The GPU is the most crucial component in the box. It will train these deep networks fast, shortening the feedback cycle.

The GPU is important is because: a) most calculations in DL are matrix operations, like matrix multiplication. They can be slow if done on the CPU. b) As we are doing thousands of these operations in a typical neural network, the slowness really adds up (as we will see in the benchmarks later). On the other hand, GPUs, rather conveniently, are able to run all these operations in parallel. They have a large number of cores, which can run an even larger number of threads. GPUs also have much higher memory bandwidth which enables them to run these parallel operations on a bunch of data at once.

Disclosure: The following are affiliate links, to help me pay for, well, more GPUs.

The choice is between a few of Nvidia’s cards: GTX 1070, GTX 1070 Ti, GTX 1080, GTX 1080 Ti and finally the Titan X. The prices might fluctuate, especially because some GPUs are great for cryptocurrency mining (wink, 1070, wink).

On performance side: GTX 1080 Ti and Titan X are similar. Roughly speaking the GTX 1080 is about 25% faster than GTX 1070. And GTX 1080 Ti is about 30% faster than GTX 1080. The new GTX 1070 Ti is very close in performance to GTX 1080.

Tim Dettmers has a great article on picking a GPU for Deep Learning, which he regularly updates as new cards come on the market.

Here are the things to consider when picking a GPU:

1. Maker: No contest on this one — get Nvidia. They have been focusing on Machine Learning for a number of years now, and it’s paying off. Their CUDA toolkit is entrenched so deeply that it is literally the only choice for the DL practitioner. Note: I wish AMD would pick up their game on this. AMD cards are cheaper and do half-precision compute at full speed.
2. Budget: The Titan X got a really bad mark here as it is offering the same performance as the 1080 Ti for about $500-$700 more. It used to be case that you could do same speed half-precision (fp16) on the old, Maxwell-based Titan X, effectively doubling your GPU memory, but not on the new one.
3. One or multiple: I considered picking a couple of 1070s (or currently 1070 Ti) instead of 1080 or 1080 Ti. This would have allowed me to either train a model on two cards or train two models at once. Currently training a model on multiple cards is a bit of a hassle, though things are changing with PyTorch and Caffe 2 offering almost linear scaling with the number of GPUs. The other option — training two models at once seemed to have more value, but I decided to get a single more powerful card now and add a second one later.
4. Memory: More memory is better. With more memory, we could deploy bigger models and use sufficiently large batch size during training (which helps the gradient flow).
5. Memory bandwidth: This enables the GPU to operate on large amounts of memory. Tim Dettmers points out that this is the most important characteristic of a GPU.

Considering all of this, I picked the GTX 1080 Ti, mainly for the training speed boost. I plan to add a second 1080 Ti soonish.

CPU

Even though the GPU is the MVP in deep learning, the CPU still matters. For example, data preparation is usually done on the CPU. The number of cores and threads per core is important if we want to parallelize all that data prep.

To stay on budget, I picked a mid-range CPU, the Intel i5 7500. It’s relatively cheap but good enough to not slow things down.

Edit: As a few people have pointed out: “probably the biggest gotcha that is unique to DL/multi-GPU is to pay attention to the PCIe lanes supported by the CPU/motherboard” (by Andrej Karpathy). We want to have each GPU have 16 PCIe lanes so it eats data as fast as possible (16 GB/s for PCIe 3.0). This means that for two cards we need 32 PCIe lanes. However, the CPU I have picked has only 16 lanes. So 2 GPUs would run in 2x8 mode (instead of 2x16). This might be a bottleneck, leading to less than ideal utilization of the graphics cards. Thus a CPU with 40 lines is recommended.

Edit 2: However, Tim Dettmers points out that having 8 lanes per card should only decrease performance by “0–10%” for two GPUs. So currently, my recommendation is: Go with 16 PCIe lanes per video card unless it gets too expensive for you. Otherwise, 8 lanes should do as well.

A good solution with to have for a double GPU machine would be an Intel Xeon processor like the E5–1620 v4 (40 PCIe lanes). Or if you want to splurge go for a higher end processor like the desktop i7–6850K.

Memory (RAM)

It’s nice to have a lot of memory if we are to be working with rather big datasets. I got 2 sticks of 16 GB, for a total of 32 GB of RAM, and plan to buy another 32 GB later.

Disk

Following Jeremy Howard’s advice, I got a fast SSD disk to keep my OS and current data on, and then a slow spinning HDD for those huge datasets (like ImageNet).
SSD: I remember when I got my first Macbook Air years ago, how blown away was I by the SSD speed. To my delight, a new generation of SSD called NVMe has made its way to market in the meantime. A 480 GB MyDigitalSSD NVMe drive was a great deal. This baby copies files at gigabytes per second.
HDD: 2 TB Seagate. While SSDs have been getting fast, HDD have been getting cheap. To somebody who has used Macbooks with 128 GB disk for the last 7 years, having this much space feels almost obscene.

Motherboard

The one thing that I kept in mind when picking a motherboard was the ability to support two GTX 1080 Ti, both in the number of PCI Express Lanes (the minimum is 2x8) and the physical size of 2 cards. Also, make sure it’s compatible with the chosen CPU. An Asus TUF Z270 did it for me.

MSI — X99A SLI PLUS should work great if you got an Intel Xeon CPU.

Power Supply

Rule of thumb: Power supply should provide enough juice for the CPU and the GPUs, plus 100 watts extra.
The Intel i5 7500 processor uses 65W, and the GPUs (1080 Ti) need 250W each, so I got a Deepcool 750W Gold PSU (currently unavailable, EVGA 750 GQ is similar). The “Gold” here refers to the power efficiency, i.e how much of the power consumed is wasted as heat.

Case

The case should be the same form factor as the motherboard. Also having enough LEDs to embarrass a Burner is a bonus.

A friend recommended the Thermaltake N23 case, which I promptly got. No LEDs sadly.

Budgeting it all in

Here is how much I spent on all the components (your costs may vary):

$700 GTX 1080 Ti
+ $190 CPU
+ $230 RAM
+ $230 SSD
+ $66 HDD
+ $130 Motherboard
+ $75 PSU
+ $50 Case
============
$1671 Total

Adding tax and fees, this nicely matches my preset budget of $1700.

This party is about to go down

---

Putting it all together

If you don’t have much experience with hardware and fear you might break something, a professional assembly might be the best option. However, this was a great learning opportunity that I couldn’t pass (even though I’ve had my share of hardware-related horror stories).

The first and important step is to read the installation manuals that came with each component. Especially important for me, as I’ve done this before once or twice, and I have just the right amount of inexperience to mess things up.

Where things go on the motherboard

Install the CPU on the Motherboard

The CPU in its slot, the lever refusing to go down.

This is done before installing the motherboard in the case. Next to the processor there is a lever that needs to be pulled up. The processor is then placed on the base (double-check the orientation). Finally, the lever comes down to fix the CPU in place.

Me being assisted in installing the CPU

.

.

But I had a quite the difficulty doing this: once the CPU was in position the lever wouldn’t go down. I actually had a more hardware-capable friend of mine video walk me through the process. Turns out the amount of force required to get the lever locked down was more than what I was comfortable with.

The installed fan

Next is fixing the fan on top of the CPU: the fan legs must be fully secured to the motherboard. Consider where the fan cable will go before installing. The processor I had came with thermal paste. If yours doesn’t, make sure to put some paste between the CPU and the cooling unit. Also, replace the paste if you take off the fan.

Install Power Supply in the Case

Fitting the power cables through the back side.

I put the Power Supply Unit (PSU) in before the motherboard to get the power cables snugly placed in case back side.

.

.

.

.

Install the Motherboard in the case

Having fun with magnets

Pretty straight forward — carefully place it and screw it in. A magnetic screwdriver was really helpful.

Then connect the power cables and the case buttons and LEDs.

.

.

Install the NVMe Disk

Just slide it in the M2 slot and screw it in. Piece of cake.

.

.

.

.

Install the RAM

The GTX 1080 Ti calmly waiting its turn as I struggle with the RAM in the background.

The memory proved quite hard to install, requiring too much effort to properly lock in. A few times I almost gave up, thinking I must be doing it wrong. Eventually one of the sticks clicked in and the other one promptly followed.

At this point, I turned the computer on to make sure it works. To my relief, it started right away!

Install the GPU

The GTX 1080 Ti setting into its new home

Finally, the GPU slid in effortlessly. 14 pins of power later and it was running.

NB: Do not plug your monitor in the external card right away. Most probably it needs drivers to function (see below).

Finally, it’s complete!

---

Software Setup

Now that we have the hardware in place, only the soft part remains. Out with the screwdriver, in with the keyboard.

Note on dual booting: If you plan to install Windows (because, you know, for benchmarks, totally not for gaming), it would be wise to do Windows first and Linux second. I didn’t and had to reinstall Ubuntu because Windows messed up the boot partition. Livewire has a detailed article on dual boot.

Install Ubuntu

Most DL frameworks are designed to work on Linux first, and eventually support other operating systems. So I went for Ubuntu, my default Linux distribution. An old 2GB USB drive was laying around and worked great for the installation. UNetbootin (OSX) or Rufus (Windows) can prepare the Linux thumb drive. The default options worked fine during the Ubuntu install.

At the time of writing, Ubuntu 17.04 was just released, so I opted for the previous version (16.04), whose quirks are much better documented online.

Ubuntu Server or Desktop: The Server and Desktop editions of Ubuntu are almost identical, with the notable exception of the visual interface (called X) not being installed with Server. I installed the Desktop and disabled autostarting X so that the computer would boot it in terminal mode. If needed, one could launch the visual desktop later by typing startx.

Getting up to date

Let’s get our install up to date. From Jeremy Howard’s excellent install-gpu script:

sudo apt-get update
sudo apt-get --assume-yes upgrade
sudo apt-get --assume-yes install tmux build-essential gcc g++ make binutils
sudo apt-get --assume-yes install software-properties-common
sudo apt-get --assume-yes install git

The Deep Learning stack

To deep learn on our machine, we need a stack of technologies to use our GPU:

• GPU driver — A way for the operating system to talk to the graphics card.
• CUDA — Allows us to run general purpose code on the GPU.
• CuDNN — Provides deep neural networks routines on top of CUDA.
• A DL framework — Tensorflow, PyTorch, Theano, etc. They make live easier by abstracting the lower levels of the stack.

Install CUDA

Download CUDA from Nvidia, or just run the code below:

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda-toolkit-8.0 

Updated to specify version 8 of CUDA. Thanks to Anurag Verma for the tip.

After CUDA has been installed the following code will add the CUDA installation to the PATH variable:

cat >> ~/.bashrc << 'EOF'
export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64\
${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
EOF
source ~/.bashrc

Now we can verify that CUDA has been installed successfully by running

nvcc --version # Checks CUDA version
nvidia-smi # Info about the detected GPUs

This should have installed the display driver as well. For me, nvidia-smi showed ERR as the device name, so I installed the latest Nvidia drivers (at the time of writing) to fix it:

wget http://us.download.nvidia.com/XFree86/Linux-x86_64/384.98/NVIDIA-Linux-x86_64-384.98.run
sudo sh NVIDIA-Linux-x86_64-384.98.run
sudo reboot

Removing CUDA/Nvidia drivers

If at any point the drivers or CUDA seem broken (as they did for me — multiple times), it might be better to start over by running:

sudo apt-get remove --purge nvidia*
sudo apt-get autoremove
sudo reboot

CuDNN

Since version 1.3 Tensorflow supports CuDNN 6, so we install that. To download CuDNN, one needs to register for a (free) developer account. After downloading, install with the following:

tar -xzf cudnn-8.0-linux-x64-v6.0.tgz
cd cuda
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/* /usr/local/cuda/include/

Anaconda

Anaconda is a great package manager for python. I’ve moved to python 3.6, so will be using the Anaconda 3 version:

wget https://repo.continuum.io/archive/Anaconda3-4.3.1-Linux-x86_64.sh -O “anaconda-install.sh”
bash anaconda-install.sh -b
cat >> ~/.bashrc << 'EOF'
export PATH=$HOME/anaconda3/bin:${PATH}
EOF
source .bashrc
conda upgrade -y --all
source activate root

Tensorflow

The popular DL framework by Google. Installation:

sudo apt install python3-pip
pip install tensorflow-gpu

Validate Tensorfow install: To make sure we have our stack running smoothly, I like to run the tensorflow MNIST example:

git clone https://github.com/tensorflow/tensorflow.git
python tensorflow/tensorflow/examples/tutorials/mnist/fully_connected_feed.py

We should see the loss decreasing during training:

Step 0: loss = 2.32 (0.139 sec)
Step 100: loss = 2.19 (0.001 sec)
Step 200: loss = 1.87 (0.001 sec)

Keras

Keras is a great high-level neural networks framework, an absolute pleasure to work with. Installation can’t be easier too:

pip install keras

PyTorch

PyTorch is a newcomer in the world of DL frameworks, but its API is modeled on the successful Torch, which was written in Lua. PyTorch feels new and exciting, mostly great, although some things are still to be implemented. We install it by running:

conda install pytorch torchvision cuda80 -c soumith

Jupyter notebook

Jupyter is a web-based IDE for Python, which is ideal for data sciency tasks. It’s installed with Anaconda, so we just configure and test it:

# Create a ~/.jupyter/jupyter_notebook_config.py with settings
jupyter notebook --generate-config
jupyter notebook --port=8888 --NotebookApp.token='' # Start it

Now if we open http://localhost:8888 we should see a Jupyter screen.

Run Jupyter on boot

Rather than running the notebook every time the computer is restarted, we can set it to autostart on boot. We will use crontab to do this, which we can edit by running crontab -e . Then add the following after the last line in the crontab file:

# Replace 'path-to-jupyter' with the actual path to the jupyter
# installation (run 'which jupyter' if you don't know it). Also
# 'path-to-dir' should be the dir where your deep learning notebooks 
# would reside (I use ~/DL/).

@reboot path-to-jupyter notebook --no-browser --port=8888 --NotebookApp.token='' --notebook-dir path-to-dir &

Outside access

I use my old trusty Macbook Air for development, so I’d like to be able to log into the DL box both from my home network, also when on the run.

SSH Key: It’s way more secure to use a SSH key to login instead of a password. Digital Ocean has a great guide on how to setup this.

SSH tunnel: If you want to access your jupyter notebook from another computer, the recommended way is to use SSH tunneling (instead of opening the notebook to the world and protecting with a password). Let’s see how we can do this:

1. First, we need an SSH server. We install it by running the following on the DL box (server):

sudo apt-get install openssh-server
sudo service ssh status

2. Then to connect over SSH tunnel, run the following script on the client:

# Replace user@host with your server user and ip.
ssh -N -f -L localhost:8888:localhost:8888 user@host

To test this, open a browser and try http://localhost:8888 from the remote machine. Your Jupyter notebook should appear.

Setup out-of-network access: Finally to access the DL box from the outside world, we need 3 things:

1. Static IP for your home network (or a service to emulate that) — so that we know on what address to connect.
2. A manual IP or a DHCP reservation giving the DL box a permanent address on your home network.
3. Port forwarding from the router to the DL box (instructions for your router).

Setting up out-of-network access depends on the router/network setup, so I’m not going into details.

---

Benchmarks

Now that we have everything running smoothly, let’s put it to the test. We’ll be comparing the newly built box to an AWS P2.xlarge instance, which is what I’ve used so far for DL. The tests are computer vision related, meaning convolutional networks with a fully connected model thrown in. We time training models on: AWS P2 instance GPU (K80), AWS P2 virtual CPU, the GTX 1080 Ti and Intel i5 7500 CPU.

MNIST Multilayer Perceptron

The “Hello World” of computer vision. The MNIST database consists of 70,000 handwritten digits. We run the Keras example on MNIST which uses Multilayer Perceptron (MLP). The MLP means that we are using only fully connected layers, not convolutions. The model is trained for 20 epochs on this dataset, which achieves over 98% accuracy out of the box.

We see that the GTX 1080 Ti is 2.4 times faster than the K80 on AWS P2 in training the model. This is rather surprising as these 2 cards should have about the same performance. I believe this is because of the virtualization or underclocking of the K80 on AWS.

The CPUs perform 9 times slower than the GPUs. As we will see later, it’s a really good result for the processors. This is due to the small model which fails to fully utilize the parallel processing power of the GPUs.

Interestingly, the desktop Intel i5–7500 achieves 2.3x speedup over the virtual CPU on Amazon.

VGG Finetuning

A VGG net will be finetuned for the Kaggle Dogs vs Cats competition. In this competition, we need to tell apart pictures of dogs and cats. Running the model on CPUs for the same number of batches wasn’t feasible. Therefore we finetune for 390 batches (1 epoch) on the GPUs and 10 batches on the CPUs. The code used is on github.

The 1080 Ti is 5.5 times faster that the AWS GPU (K80). The difference in the CPUs performance is about the same as the previous experiment (i5 is 2.6x faster). However, it’s absolutely impractical to use CPUs for this task, as the CPUs were taking ~200x more time on this large model that includes 16 convolutional layers and a couple semi-wide (4096) fully connected layers on top.

Wasserstein GAN

A GAN (Generative adversarial network) is a way to train a model to generate images. GAN achieves this by pitting two networks against each other: A Generator which learns to create better and better images, and a Discriminator that tries to tell which images are real and which are dreamt up by the Generator.

The Wasserstein GAN is an improvement over the original GAN. We will use a PyTorch implementation, that is very similar to the one by the WGAN author. The models are trained for 50 steps, and the loss is all over the place which is often the case with GANs. CPUs aren’t considered.

The GTX 1080 Ti finishes 5.5x faster than the AWS P2 K80, which is in line with the previous results.

Style Transfer

The final benchmark is on the original Style Transfer paper (Gatys et al.), implemented on Tensorflow (code available). Style Transfer is a technique that combines the style of one image (a painting for example) and the content of another image. Check out my previous post for more details on how Style Transfer works.

The GTX 1080 Ti outperforms the AWS K80 by a factor of 4.3. This time the CPUs are 30-50 times slower than graphics cards. The slowdown is less than on the VGG Finetuning task but more than on the MNIST Perceptron experiment. The model uses mostly the earlier layers of the VGG network, and I suspect this was too shallow to fully utilize the GPUs.

---

The DL box is in the next room and a large model is training on it. Was it a wise investment? Time will tell but it is beautiful to watch the glowing LEDs in the dark and to hear its quiet hum as models are trying to squeeze out that extra accuracy percentage point.