「PyTorch」：1-PyTorch Explained

PyTorch框架学习。

本篇文章主要介绍PyTorch的相关背景知识。

PyTorch Explained: Python DNN API

【PyTorch，是使用Python来深度学习的一个API。】

PyTorch torch.Tensor objects that are created from NumPy ndarray objects, share memory. This makes the transition between PyTorch and NumPy very cheap from a performance perspective.

With PyTorch tensors, GPU support is built-in. It’s very easy with PyTorch to move tensors to and from a GPU if we have one installed on our system.

【PyTorch原生支持GPU】

A Brief History

The initial release of PyTorch was in October of 2016, and before PyTorch was created, there was and still is, another framework called Torch. Torch is a machine learning framework that’s been around for quite a while and is based on the Lua programming language.

The connection between PyTorch and this Lua version, called Torch, exists because many of the developers who maintain the Lua version are the individuals who created PyTorch.

PyTorch is that it was created and is maintained by Facebook. This is because Soumith Chintala worked at Facebook AI Research when PyTorch was created (still does at the time of this writing). However, there are many other companies with a vested interest in PyTorch.

Deep Learning With PyTorch

PyTorch的包和主要组成：

Package	Description
torch	PyTorch的顶层包和tensor库
torch.nn	包含构建NN的模型和扩展类
torch.autograd	PyTorch中支持的Tensor操作
torch.nn.functional	包含构建NN的函数接口，像loss function, activation fucntion, convolution operation
torch.optim	包含标准的优化，像SGD, Adam
torch.utils	包含实用类，像数据集，数据装载器，方便数据预处理
torchvision	提供著名的数据集，模型架构和计算机视觉图像转换

Why use PyTorch for Deep Learning ?

• PyTorch is thin and stays out of the way!
• PyTorch is as close as it gets to the real thing!

Investing In PyTorch As A Deep Learning Framework

To optimize neural networks, we need to calculate derivatives, and to do this computationally, deep learning frameworks use what are called computational graphs.

计算图。

Computational graphs are used to graph the function operations that occur on tensors inside neural networks.

计算图用来绘制NN中内部张量发生的函数运算。

These graphs are then used to compute the derivatives needed to optimize the neural network. PyTorch uses a computational graph that is called a dynamic computational graph. This means that the graph is generated on the fly as the operations are created.

计算图用来计算偏微分优化NN。PyTorch使用的是动态计算图，图是即时创建的。

CUDA Explained - Why Deep Learning Uses GPUs

In this post, we are going to introduce CUDA at a high-level.

The goal of this post is to help beginners understand what CUDA is and how it fits in with PyTorch, and more importantly, why we even use GPUs in neural network programming anyway.

Graphics Processing Unit(GPU)

To understand CUDA, we need to have a working knowledge of graphics processing units (GPUs). A GPU is a processor that is good at handling specialized computations.

【GPU擅长处理某种特定的计算】

This is in contrast to a central processing unit (CPU), which is a processor that is good at handling general computations.

【CPU擅长处理general computations.】

Parallel Computing

Parallel computing is a type of computation where by a particular computation is broken into independent smaller computations that can be carried out simultaneously. The resulting computations are then recombined, or synchronized, to form the result of the original larger computation.

Neural Networks Are Embarrassingly Parallel

In parallel computing, an embarrassingly parallel task is one where little or no effort is needed to separate the overall task into a set of smaller tasks to be computed in parallel.

Tasks that embarrassingly parallel are ones where it’s easy to see that the set of smaller tasks are independent with respect to each other.

Nvidia Hardware(GPU) And Software(CUDA)

Nvidia is a technology company that designs GPUs, and they have created CUDA as a software platform that pairs with their GPU hardware making it easier for developers to build software that accelerates computations using the parallel processing power of Nvidia GPUs.

An Nvidia GPU is the hardware that enables parallel computations, while CUDA is a software layer that provides an API for developers.

GPU是能平行运算的硬件，而CUDA是为开发者提供的上层API。

PyTorch Comes With CUDA

One of the benefits of using PyTorch, or any other neural network API is that parallelism comes baked into the API. This means that as neural network programmers, we can focus more on building neural networks and less on performance issues.

关注怎么构建NN，而不是一下performance issues.

Now, if we wanted to work on the PyTorch core development team or write PyTorch extensions, it would probably be useful to know how to use CUDA directly.

After all, PyTorch is written in all of these:

• Python
• C++
• CUDA