This article is for many Colab official Welcome summarize series, artificial intelligence and depth of learning more knowledge of reference of artificial intelligence and depth of learning practical https://github.com/wx-chevalier/AIDL-Series series.

Colaboratory

Colaboratory is a free Jupyter notebook environment, does not require any settings you can use, and runs entirely in the cloud. With Colaboratory, we can write in the browser and executing code, save and share analysis results, and the use of powerful computing resources, GPU included with TPU to run our test code.

Colab can easily link with Google Driver and Github, we can use the Open in Colab plug quickly open Notebook on Github, or using similar colab.research.google.com/github/goog... this link to open. If you need to save the Notebook back to Github, direct use File→Save a copy to GitHubcan be. For example I all associated Colab disposed in the return code the AIDL-Workbench / Colab .

When dependence and run

Reliance installation

Colab dependence provides a convenient mounting feature that allows pip or apt-get command to install:

# Importing a library that is not in Colaboratory
!pip install -q matplotlib-venn
!apt-get -qq install -y libfluidsynth1

# Upgrading TensorFlow
# To determine which version you're using:
!pip show tensorflow

# For the current version:
!pip install --upgrade tensorflow

# For a specific version:
!pip install tensorflow==1.2

# For the latest nightly build:
!pip install tf-nightly

# Install Pytorch
from os import path
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())

accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'

!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.0-{platform}-linux_x86_64.whl torchvision

# Install 7zip reader libarchive
# https://pypi.python.org/pypi/libarchive
!apt-get -qq install -y libarchive-dev && pip install -q -U libarchive
import libarchive

# Install GraphViz & PyDot
# https://pypi.python.org/pypi/pydot
!apt-get -qq install -y graphviz && pip install -q pydot
import pydot

# Install cartopy
!apt-get -qq install python-cartopy python3-cartopy
import cartopy
复制代码

In Colab you can also set the environment variable:

%env KAGGLE_USERNAME=abcdefgh
复制代码

Hardware Acceleration

We can see the hardware Colab provided us by the following ways:

from tensorflow.python.client import device_lib
device_lib.list_local_devices()

!ls /proc
# CPU信息
!cat /proc/cpuinfo
# 内存
!cat /proc/meminfo
# 版本
!cat /proc/version
# 设备
!cat /proc/devices
# 空间
!df
复制代码

If support is required to start GPU Notebook: Click Edit->notebook settings->hardware accelerator->GPU, and then determines whether a code GPU available devices:

import tensorflow as tf
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
复制代码

We can compare the differences in the GPU and CPU on CNN convolution operation by layer to build a classic:

import tensorflow as tf
import timeit

# See https://www.tensorflow.org/tutorials/using_gpu#allowing_gpu_memory_growth
config = tf.ConfigProto()
config.gpu_options.allow_growth = True

with tf.device('/cpu:0'):
  random_image_cpu = tf.random_normal((100, 100, 100, 3))
  net_cpu = tf.layers.conv2d(random_image_cpu, 32, 7)
  net_cpu = tf.reduce_sum(net_cpu)

with tf.device('/gpu:0'):
  random_image_gpu = tf.random_normal((100, 100, 100, 3))
  net_gpu = tf.layers.conv2d(random_image_gpu, 32, 7)
  net_gpu = tf.reduce_sum(net_gpu)

sess = tf.Session(config=config)

# Test execution once to detect errors early.
try:
  sess.run(tf.global_variables_initializer())
except tf.errors.InvalidArgumentError:
  print(
      '\n\nThis error most likely means that this notebook is not '
      'configured to use a GPU.  Change this in Notebook Settings via the '
      'command palette (cmd/ctrl-shift-P) or the Edit menu.\n\n')
  raise

def cpu():
  sess.run(net_cpu)

def gpu():
  sess.run(net_gpu)

# Runs the op several times.
print('Time (s) to convolve 32x7x7x3 filter over random 100x100x100x3 images '
      '(batch x height x width x channel). Sum of ten runs.')
print('CPU (s):')
cpu_time = timeit.timeit('cpu()', number=10, setup="from __main__ import cpu")
print(cpu_time)
print('GPU (s):')
gpu_time = timeit.timeit('gpu()', number=10, setup="from __main__ import gpu")
print(gpu_time)
print('GPU speedup over CPU: {}x'.format(int(cpu_time/gpu_time)))

sess.close()
复制代码

Running locally

Colab also supports direct connection to a local Jupyter Notebook server to run, you first need to enable jupyter_http_over_ws extension:

pip install jupyter_http_over_ws
jupyter serverextension enable --py jupyter_http_over_ws
复制代码

Then start Jupyter server in the normal way, set a flag to indicate clearly WebSocket trust from Colaboratory front of the connection:

jupyter notebook \
  --NotebookApp.allow_origin='https://colab.research.google.com' \
  --port=8888 \
  --NotebookApp.port_retries=0
复制代码

Then select the program code is executed to the local connection to the Colab the Notebook.

Data with an external module

Colab in the notebook and py are based on the default file / content / as working directory, execute the following command to manually switch the working directory, for example:

import os

path = "/content/drive/colab-notebook/lesson1-week2/assignment2"
os.chdir(path)
os.listdir(path)
复制代码

Google Driver

, A lot of training and test data acquired at the time of the last experiment, the storage and loading has been a headache problem; in Colab, I will Awesome DataSets https://url.wx-coder.cn/FqwyP ) in related data AIDL-Workbench / datasets script in persistent storage in Google Driver in.

In Colab we can Google Driver mount when the working path:

from google.colab import drive
drive.mount("/content/drive")

print('Files in Drive:')
!ls /content/drive/'My Drive'
复制代码

And then to create operational through normal Linux Shell command:

# Working with files
# Create directories for the new project
!mkdir -p drive/kaggle/talkingdata-adtracking-fraud-detection

!mkdir -p drive/kaggle/talkingdata-adtracking-fraud-detection/input/train
!mkdir -p drive/kaggle/talkingdata-adtracking-fraud-detection/input/test
!mkdir -p drive/kaggle/talkingdata-adtracking-fraud-detection/input/valid

# Download files
!wget -O /content/drive/'My Drive'/Data/fashion_mnist/train-images-idx3-ubyte.gz http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz

# Download and Unzip files
%env DIR=/content/drive/My Drive/Data/animals/cats_and_dogs

!rm -rf "$DIR"
!mkdir -pv "$DIR"
!wget -O "$DIR"/Cat_Dog_data.zip https://s3.amazonaws.com/content.udacity-data.com/nd089/Cat_Dog_data.zip

# remove existing directories
!(cd "$DIR" && unzip -qqj Cat_Dog_data.zip -d .)
复制代码

External Python file

Colab Python allows us to upload files to the working directory, or load the Google Driver in Python:

# Import modules
import imp
helper = imp.new_module('helper')
exec(open("drive/path/to/helper.py").read(), helper.__dict__)

fc_model = imp.new_module('fc_model')
exec(open("pytorch-challenge/deep-learning-v2-pytorch/intro-to-pytorch/fc_model.py").read(), fc_model.__dict__)
复制代码

File upload and download

Colab also allows us to upload directly from the local file when running the script, or download the generated model to a local file:

from google.colab import files

# Upload file
uploaded = files.upload()

for fn in uploaded.keys():
  print('User uploaded file "{name}" with length {length} bytes'.format(
      name=fn, length=len(uploaded[fn])))

# Download file
with open('example.txt', 'w') as f:
  f.write('some content')

files.download('example.txt')
复制代码

BigQuery

If we use the BigQuery provides a large data query and management functions, then in Colab also be introduced directly into the data sources in BigQuery:

from google.cloud import bigquery

client = bigquery.Client(project=project_id)

sample_count = 2000
row_count = client.query('''
  SELECT
    COUNT(*) as total
  FROM `bigquery-public-data.samples.gsod`''').to_dataframe().total[0]

df = client.query('''
  SELECT
    *
  FROM
    `bigquery-public-data.samples.gsod`
  WHERE RAND() < %d/%d
''' % (sample_count, row_count)).to_dataframe()

print('Full dataset has %d rows' % row_count)
复制代码

Use the controls

grid

Colab and provides Grid Tab control for us, for us to build a simple chart layout:

import numpy as np
import random
import time
from matplotlib import pylab
grid = widgets.Grid(2, 2)
for i in range(20):
  with grid.output_to(random.randint(0, 1), random.randint(0, 1)):
    grid.clear_cell()
    pylab.figure(figsize=(2, 2))
    pylab.plot(np.random.random((10, 1)))
  time.sleep(0.5)
复制代码

TabBar offers tab of the layout:

from __future__ import print_function

from google.colab import widgets
from google.colab import output
from matplotlib import pylab
from six.moves import zip


def create_tab(location):
  tb = widgets.TabBar(['a', 'b'], location=location)
  with tb.output_to('a'):
    pylab.figure(figsize=(3, 3))
    pylab.plot([1, 2, 3])
  # Note you can access tab by its name (if they are unique), or
  # by its index.
  with tb.output_to(1):
    pylab.figure(figsize=(3, 3))
    pylab.plot([3, 2, 3])
    pylab.show()


print('Different orientations for tabs')

positions = ['start', 'bottom', 'end', 'top']

for p, _ in zip(positions, widgets.Grid(1, 4)):
  print('---- %s ---' % p)
  create_tab(p)
复制代码

Forms

It is commendable that, Colab also provides form-based interactive components, to facilitate the application we can build dynamic input:

#@title String fields

text = 'value' #@param {type:"string"}
dropdown = '1st option' #@param ["1st option", "2nd option", "3rd option"]
text_and_dropdown = '2nd option' #@param ["1st option", "2nd option", "3rd option"] {allow-input: true}

print(text)
print(dropdown)
print(text_and_dropdown)
复制代码

Fast deep learning in practice in Google Colab