pdf

艾伯特AlexNet2012[aibbt.com]

  • 1星
  • 日期: 2018-11-07
  • 大小: 1.67MB
  • 所需积分:1分
  • 下载次数:0
  • favicon收藏
  • rep举报
  • 分享
  • free评论
标签: 电子

艾伯特AlexNet2012[aibbt.com]

文档内容节选

ImageNet Classication with Deep Convolutional Neural Networks Alex Krizhevsky University of Toronto krizcsutorontoca Ilya Sutskever University of Toronto ilyacsutorontoca Geoffrey E Hinton University of Toronto hintoncsutorontoca Abstract We trained a large deep convolutional neural network to classify the 12 million highresolution images in the ImageNet LSVRC2010 contest into the 1000 dif ferent classes On the test data we achieved top1 and top5 error rates of 375 and 170 which is considerably ......

ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky University of Toronto kriz@cs.utoronto.ca Ilya Sutskever University of Toronto ilya@cs.utoronto.ca Geoffrey E. Hinton University of Toronto hinton@cs.utoronto.ca Abstract We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif- ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make train- ing faster, we used non-saturating neurons and a very efficient GPU implemen- tation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. 1 Introduction Current approaches to object recognition make essential use of machine learning methods. To im- prove their performance, we can collect larger datasets, learn more powerful models, and use bet- ter techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current- best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to col- lect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories. To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this prob- lem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be con- trolled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse. 1 Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luck- ily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting. The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1. Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance. In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available. 2 The Dataset ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Ama- zon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images. ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model. ImageNet consists of variable-resolution images, while our system requires a constant input dimen- sionality. Therefore, we down-sampled the images to a fixed resolution of 256 ⇥ 256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256⇥256 patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels. 3 The Architecture The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first. 1http://code.google.com/p/cuda-convnet/ 2 3.1 ReLU Nonlinearity The standard way to model a neuron’s output f as a function of its input x is with f (x) = tanh(x) or f (x) = (1 + ex)1. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f (x) = max(0, x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural net- works with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations re- quired to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional net- work. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models. We are not the first to consider alternatives to tradi- tional neuron models in CNNs. For example, Jarrett et al. [11] claim that the nonlinearity f (x) = |tanh(x)| works particularly well with their type of contrast nor- malization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the pri- mary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when us- ing ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets. 3.2 Training on Multiple GPUs Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rates for each net- work were chosen independently to make train- ing as fast as possible. No regularization of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consis- tently learn several times faster than equivalents with saturating neurons. A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation. The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cire¸san et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2. 2The one-GPU net actually has the same number of kernels as the two-GPU net in the final convolutional layer. This is because most of the net’s parameters are in the first fully-connected layer, which takes the last convolutional layer as input. So to make the two nets have approximately the same number of parameters, we did not halve the size of the final convolutional layer (nor the fully-conneced layers which follow). Therefore this comparison is biased in favor of the one-GPU net, since it is bigger than “half the size” of the two-GPU net. 3 3.3 Local Response Normalization ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by ai x,y the activity of a neuron computed by applying kernel i at position (x, y) and then applying the ReLU nonlinearity, the response-normalized activity bi x,y is given by the expression bi x,y = ai x,y/0@k + ↵ min(N1,i+n/2)Xj=max(0,in/2) (aj x,y)21A where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k, n, ↵, and are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, ↵ = 104, and = 0.75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5). This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization3. 3.4 Overlapping Pooling Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z ⇥ z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2, which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit. 3.5 Overall Architecture Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fully- connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution. The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fully- connected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer. The first convolutional layer filters the 224⇥ 224⇥ 3 input image with 96 kernels of size 11⇥ 11⇥ 3 with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring 3We cannot describe this network in detail due to space constraints, but it is specified precisely by the code and parameter files provided here: http://code.google.com/p/cuda-convnet/. 4 Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000. neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5 ⇥ 5 ⇥ 48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 ⇥ 3 ⇥ 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3 ⇥ 3 ⇥ 192 , and the fifth convolutional layer has 256 kernels of size 3 ⇥ 3 ⇥ 192. The fully-connected layers have 4096 neurons each. 4 Reducing Overfitting Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting. 4.1 Data Augmentation The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free. The first form of data augmentation consists of generating image translations and horizontal reflec- tions. We do this by extracting random 224⇥ 224 patches (and their horizontal reflections) from the 256⇥256 images and training our network on these extracted patches4. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly inter- dependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five 224 ⇥ 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches. The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, 4This is the reason why the input images in Figure 2 are 224 ⇥ 224 ⇥ 3-dimensional. 5
更多简介内容

推荐帖子

今天上午10:00 有奖直播:深入探讨迈来芯高可靠可编程霍尔开关,如何满足客户个性...
今天上午10:00 有奖直播:深入探讨迈来芯高可靠可编程霍尔开关,如何满足客户个性开发的需求   >>点击进入直播   直播时间:5月21日上午10:00-11:30   直播主题: 深入探讨迈来芯高可靠可编程霍尔开关,如何满足客户个性开发的需求 直播介绍: 迈来芯的霍尔开关及锁存器产品拥有26年的丰富经验,累计销量超过20亿片。目前我们的霍尔
EEWORLD社区 综合技术交流
CMOS的正常工作电流下CMOS电路设计问题
    这次遇到的问题是这样的,由于LDO的电源输出能力有限,同时也是保证可靠性,某些保护电路必须直接电源上,这里需要注释的是,LIMP HOME“跛行回家”即使我们的5V系统出现了问题,我们也必须保证部分重要的功能能继续工作,因此为了满足这个条件,关键信号的供电只能是直接接电池。     这次第一个出问题的是一个CMOS的与门(HE4000B系列的)。 我们的控制信号出自MCU(5V
火辣西米秀 【模拟与混合信号】
采用分级字线结构可提高SRAM读写速度及降低电路动态功耗
  采用分级字线结构的存储器将整个存储阵列划分为若干个相同的子阵列。与非分级字线结构相比,它需要采用多级的字线译码才能完成对存储单元的寻址。如图1所示,整个电路采用层次化字线的多分割阵列结构。     图1 分级字线结构   采用层次化字线分割结构不仅能提高工作速度,而且能大大降低功耗。这是因为字线分割结构使原本同时被激活的存储单元变为只有被选中的块内的存储单元
是酒窝啊 【其他嵌入式操作系统】
继电器触点保护电路、开关电源RCD电路、MOS管构造几个问题 1.继电器线圈与RC串联电...
继电器触点保护电路、开关电源RCD电路、MOS管构造几个问题 1.继电器线圈与RC串联电路并联,为了吸收线圈断电时的自感电流。这个和开关电源的初级RCD吸收电路有没有什么相同之处?2.三极管是两个PN结构成的,那么MoS管呢?MOS管也可以看成两个二极管的构造。为什么三极管是电流控制器件,而MOS管是电压控制器件? 3.场效应管分为结型场效应管(JFET)和绝缘栅场效应管(FET)(M
QWE4562009 【电路观察室】
Modbus多机通信程序设计
    给从机下发不同的指令,从机去执行不同的操作,这个就是判断一下功能码即可,和我们前边学的实用串口例程是类似的。多机通信,无非就是添加了一个设备地址判断而已,难度也不大。我们找了一个 Modbus 调试精灵,通过设置设备地址,读写寄存器的地址以及数值数量等参数,可以直接替代串口调试助手,比较方便的下发多个字节的数据,如图 18-7 所示。我们先来就图中的设置和数据来对 Modbus 做进一步的
Jacktang 【微控制器 MCU】
工程人员安全黑匣子的方案设计
前面几个帖子分别介绍了SensorTileBox的使用,APP的连接使用。通过编程模式测试环境传感器的编程使用,活动识别的配置使用。 SensorTile.box的物联网相关应用(如何连上Microsoft Azure IoT云服务) http://bbs.eeworld.com.cn/thread-1118522-1-1.html   SensorTile.box的物联网
sipower ST MEMS传感器创意设计大赛专区

评论

登录/注册

意见反馈

求资源

回顶部

datasheet推荐 换一换

About Us 关于我们 客户服务 联系方式 器件索引 网站地图 最新更新 手机版 版权声明

北京市海淀区知春路23号集成电路设计园量子银座1305 电话:(010)82350740 邮编:100191

电子工程世界版权所有 京ICP证060456号 京ICP备10001474号 电信业务审批[2006]字第258号函 京公海网安备110108001534 Copyright © 2005-2020 EEWORLD.com.cn, Inc. All rights reserved
$(function(){ var appid = $(".select li a").data("channel"); $(".select li a").click(function(){ var appid = $(this).data("channel"); $('.select dt').html($(this).html()); $('#channel').val(appid); }) })