「PyTorch」:1-PyTorch Explained



PyTorch Explained: Python DNN API


PyTorch torch.Tensor objects that are created from NumPy ndarray objects, share memory. This makes the transition between PyTorch and NumPy very cheap from a performance perspective.

With PyTorch tensors, GPU support is built-in. It’s very easy with PyTorch to move tensors to and from a GPU if we have one installed on our system.


A Brief History

The initial release of PyTorch was in October of 2016, and before PyTorch was created, there was and still is, another framework called Torch. Torch is a machine learning framework that’s been around for quite a while and is based on the Lua programming language.

The connection between PyTorch and this Lua version, called Torch, exists because many of the developers who maintain the Lua version are the individuals who created PyTorch.

PyTorch is that it was created and is maintained by Facebook. This is because Soumith Chintala worked at Facebook AI Research when PyTorch was created (still does at the time of this writing). However, there are many other companies with a vested interest in PyTorch.

Deep Learning With PyTorch


Package Description
torch PyTorch的顶层包和tensor库
torch.nn 包含构建NN的模型和扩展类
torch.autograd PyTorch中支持的Tensor操作
torch.nn.functional 包含构建NN的函数接口,像loss function, activation fucntion, convolution operation
torch.optim 包含标准的优化,像SGD, Adam
torch.utils 包含实用类,像数据集,数据装载器,方便数据预处理
torchvision 提供著名的数据集,模型架构和计算机视觉图像转换

Why use PyTorch for Deep Learning ?

  • PyTorch is thin and stays out of the way!
  • PyTorch is as close as it gets to the real thing!

Investing In PyTorch As A Deep Learning Framework

To optimize neural networks, we need to calculate derivatives, and to do this computationally, deep learning frameworks use what are called computational graphs.


Computational graphs are used to graph the function operations that occur on tensors inside neural networks.


These graphs are then used to compute the derivatives needed to optimize the neural network. PyTorch uses a computational graph that is called a dynamic computational graph. This means that the graph is generated on the fly as the operations are created.


CUDA Explained - Why Deep Learning Uses GPUs

In this post, we are going to introduce CUDA at a high-level.

The goal of this post is to help beginners understand what CUDA is and how it fits in with PyTorch, and more importantly, why we even use GPUs in neural network programming anyway.

Graphics Processing Unit(GPU)

To understand CUDA, we need to have a working knowledge of graphics processing units (GPUs). A GPU is a processor that is good at handling specialized computations.


This is in contrast to a central processing unit (CPU), which is a processor that is good at handling general computations.

【CPU擅长处理general computations.】

Parallel Computing

Parallel computing is a type of computation where by a particular computation is broken into independent smaller computations that can be carried out simultaneously. The resulting computations are then recombined, or synchronized, to form the result of the original larger computation.

Neural Networks Are Embarrassingly Parallel

In parallel computing, an embarrassingly parallel task is one where little or no effort is needed to separate the overall task into a set of smaller tasks to be computed in parallel.

Tasks that embarrassingly parallel are ones where it’s easy to see that the set of smaller tasks are independent with respect to each other.

Nvidia Hardware(GPU) And Software(CUDA)

Nvidia is a technology company that designs GPUs, and they have created CUDA as a software platform that pairs with their GPU hardware making it easier for developers to build software that accelerates computations using the parallel processing power of Nvidia GPUs.

An Nvidia GPU is the hardware that enables parallel computations, while CUDA is a software layer that provides an API for developers.


PyTorch Comes With CUDA

One of the benefits of using PyTorch, or any other neural network API is that parallelism comes baked into the API. This means that as neural network programmers, we can focus more on building neural networks and less on performance issues.

关注怎么构建NN,而不是一下performance issues.

Now, if we wanted to work on the PyTorch core development team or write PyTorch extensions, it would probably be useful to know how to use CUDA directly.

After all, PyTorch is written in all of these:

  • Python
  • C++
  • CUDA

「PyTorch」:1-PyTorch Explained




Posted on


Updated on


Licensed under