.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "build/examples_action_recognition/demo_tsn_ucf101.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_build_examples_action_recognition_demo_tsn_ucf101.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_build_examples_action_recognition_demo_tsn_ucf101.py:

1. Getting Started with Pre-trained TSN Models on UCF101
===========================================================

`UCF101 <https://www.crcv.ucf.edu/data/UCF101.php>`_  is an action recognition dataset
of realistic action videos, collected from YouTube. With 13,320 short trimmed videos
from 101 action categories, it is one of the most widely used dataset in the research
community for benchmarking state-of-the-art video action recognition models.

`TSN <https://arxiv.org/abs/1608.00859>`_ (Temporal Segment Network) is a widely adopted video
classification method. It is proposed to incorporate temporal information from an entire video.
The idea is straightforward: we can evenly divide the video into several segments,
process each segment individually, obtain segmental consensus from each segment, and perform
final prediction. TSN is more like a general algorithm, rather than a specific network architecture.
It can work with both 2D and 3D neural networks.

In this tutorial, we will demonstrate how to load a pre-trained TSN model from :ref:`gluoncv-model-zoo`
and classify video frames from the Internet or your local disk into one of the 101 action classes.

Step by Step
------------

We will show two exmaples here. For simplicity, we first try out a pre-trained UCF101 model
on a single video frame. This is actually an image action recognition problem.

First, please follow the `installation guide <../../index.html#installation>`__
to install ``MXNet`` and ``GluonCV`` if you haven't done so yet.

.. GENERATED FROM PYTHON SOURCE LINES 28-38

.. code-block:: default


    import matplotlib.pyplot as plt
    import numpy as np
    import mxnet as mx
    from mxnet import gluon, nd, image
    from mxnet.gluon.data.vision import transforms
    from gluoncv.data.transforms import video
    from gluoncv import utils
    from gluoncv.model_zoo import get_model








.. GENERATED FROM PYTHON SOURCE LINES 39-40

Then, we download and show the example image:

.. GENERATED FROM PYTHON SOURCE LINES 41-50

.. code-block:: default


    url = 'https://github.com/bryanyzhu/tiny-ucf101/raw/master/ThrowDiscus.png'
    im_fname = utils.download(url)

    img = image.imread(im_fname)

    plt.imshow(img.asnumpy())
    plt.show()




.. image-sg:: /build/examples_action_recognition/images/sphx_glr_demo_tsn_ucf101_001.png
   :alt: demo tsn ucf101
   :srcset: /build/examples_action_recognition/images/sphx_glr_demo_tsn_ucf101_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    Downloading ThrowDiscus.png from https://github.com/bryanyzhu/tiny-ucf101/raw/master/ThrowDiscus.png...

      0%|          | 0/147 [00:00<?, ?KB/s]
    148KB [00:00, 42543.83KB/s]            




.. GENERATED FROM PYTHON SOURCE LINES 51-54

In case you don't recognize it, the image is a man throwing discus. :)

Now we define transformations for the image.

.. GENERATED FROM PYTHON SOURCE LINES 54-61

.. code-block:: default


    transform_fn = transforms.Compose([
        video.VideoCenterCrop(size=224),
        video.VideoToTensor(),
        video.VideoNormalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])








.. GENERATED FROM PYTHON SOURCE LINES 62-68

This transformation function does three things:
center crop the image to 224x224 in size,
transpose it to ``num_channels*height*width``,
and normalize with mean and standard deviation calculated across all ImageNet images.

What does the transformed image look like?

.. GENERATED FROM PYTHON SOURCE LINES 68-73

.. code-block:: default


    img_list = transform_fn([img.asnumpy()])
    plt.imshow(np.transpose(img_list[0], (1,2,0)))
    plt.show()




.. image-sg:: /build/examples_action_recognition/images/sphx_glr_demo_tsn_ucf101_002.png
   :alt: demo tsn ucf101
   :srcset: /build/examples_action_recognition/images/sphx_glr_demo_tsn_ucf101_002.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 74-78

Can't recognize anything? *Don't panic!* Neither do I.
The transformation makes it more "model-friendly", instead of "human-friendly".

Next, we load a pre-trained VGG16 model. The VGG16 model is trained using TSN with three segments.

.. GENERATED FROM PYTHON SOURCE LINES 78-81

.. code-block:: default


    net = get_model('vgg16_ucf101', nclass=101, pretrained=True)





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    Downloading /root/.mxnet/models/vgg16-e660d456.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/vgg16-e660d456.zip...

      0%|          | 0/500159 [00:00<?, ?KB/s]
      0%|          | 102/500159 [00:00<10:08, 822.39KB/s]
      0%|          | 508/500159 [00:00<03:42, 2242.36KB/s]
      0%|          | 2189/500159 [00:00<01:07, 7370.77KB/s]
      2%|1         | 7737/500159 [00:00<00:20, 23881.16KB/s]
      3%|2         | 14941/500159 [00:00<00:12, 39717.30KB/s]
      5%|4         | 23243/500159 [00:00<00:08, 53559.31KB/s]
      6%|6         | 30519/500159 [00:00<00:07, 59574.23KB/s]
      8%|7         | 38771/500159 [00:00<00:06, 66346.25KB/s]
      9%|9         | 47002/500159 [00:00<00:06, 71219.66KB/s]
     11%|#         | 54249/500159 [00:01<00:07, 60878.94KB/s]
     12%|#2        | 62376/500159 [00:01<00:06, 66369.23KB/s]
     14%|#3        | 70015/500159 [00:01<00:06, 69153.14KB/s]
     16%|#5        | 78414/500159 [00:01<00:05, 73369.37KB/s]
     17%|#7        | 86106/500159 [00:01<00:05, 74393.38KB/s]
     19%|#8        | 94127/500159 [00:01<00:05, 75654.78KB/s]
     20%|##        | 102288/500159 [00:01<00:05, 77398.70KB/s]
     22%|##2       | 110236/500159 [00:01<00:05, 77659.49KB/s]
     24%|##3       | 118677/500159 [00:01<00:04, 79652.40KB/s]
     25%|##5       | 126683/500159 [00:02<00:04, 78653.01KB/s]
     27%|##6       | 134841/500159 [00:02<00:04, 79473.76KB/s]
     29%|##8       | 142888/500159 [00:02<00:04, 79761.82KB/s]
     30%|###       | 150881/500159 [00:02<00:04, 79466.58KB/s]
     32%|###1      | 159188/500159 [00:02<00:04, 80536.33KB/s]
     33%|###3      | 167251/500159 [00:02<00:04, 79463.19KB/s]
     35%|###5      | 175502/500159 [00:02<00:04, 80363.36KB/s]
     37%|###6      | 183546/500159 [00:02<00:03, 79353.28KB/s]
     38%|###8      | 191503/500159 [00:02<00:03, 79415.10KB/s]
     40%|###9      | 199733/500159 [00:02<00:03, 80266.61KB/s]
     42%|####1     | 207765/500159 [00:03<00:03, 79916.81KB/s]
     43%|####3     | 216026/500159 [00:03<00:03, 80586.48KB/s]
     45%|####4     | 224088/500159 [00:03<00:03, 79786.19KB/s]
     46%|####6     | 232091/500159 [00:03<00:03, 79857.17KB/s]
     48%|####8     | 240126/500159 [00:03<00:03, 80002.08KB/s]
     50%|####9     | 248128/500159 [00:03<00:03, 79393.54KB/s]
     51%|#####1    | 256414/500159 [00:03<00:03, 80421.37KB/s]
     53%|#####2    | 264459/500159 [00:03<00:02, 79576.44KB/s]
     54%|#####4    | 272420/500159 [00:03<00:02, 79223.13KB/s]
     56%|#####6    | 280875/500159 [00:03<00:02, 80799.21KB/s]
     58%|#####7    | 288959/500159 [00:04<00:02, 79796.98KB/s]
     59%|#####9    | 297093/500159 [00:04<00:02, 80241.96KB/s]
     61%|######1   | 305121/500159 [00:04<00:02, 79417.63KB/s]
     63%|######2   | 313067/500159 [00:04<00:02, 79363.78KB/s]
     64%|######4   | 321006/500159 [00:04<00:02, 73621.50KB/s]
     66%|######5   | 328449/500159 [00:04<00:02, 71644.60KB/s]
     67%|######7   | 337042/500159 [00:04<00:02, 75659.26KB/s]
     69%|######8   | 344676/500159 [00:04<00:02, 75509.68KB/s]
     71%|#######   | 353133/500159 [00:04<00:01, 78129.81KB/s]
     72%|#######2  | 361012/500159 [00:05<00:01, 78320.10KB/s]
     74%|#######3  | 368875/500159 [00:05<00:01, 78131.49KB/s]
     75%|#######5  | 377456/500159 [00:05<00:01, 80398.89KB/s]
     77%|#######7  | 385514/500159 [00:05<00:01, 79246.65KB/s]
     79%|#######8  | 393522/500159 [00:05<00:01, 79229.79KB/s]
     80%|########  | 401952/500159 [00:05<00:01, 80726.96KB/s]
     82%|########1 | 410035/500159 [00:05<00:01, 79028.49KB/s]
     84%|########3 | 418403/500159 [00:05<00:01, 80284.26KB/s]
     85%|########5 | 426444/500159 [00:05<00:00, 79842.60KB/s]
     87%|########6 | 434437/500159 [00:05<00:00, 79308.67KB/s]
     89%|########8 | 442749/500159 [00:06<00:00, 80432.77KB/s]
     90%|######### | 450799/500159 [00:06<00:00, 79286.61KB/s]
     92%|#########1| 458835/500159 [00:06<00:00, 79297.79KB/s]
     93%|#########3| 467224/500159 [00:06<00:00, 80652.63KB/s]
     95%|#########5| 475295/500159 [00:06<00:00, 79412.27KB/s]
     97%|#########6| 483796/500159 [00:06<00:00, 81057.99KB/s]
     98%|#########8| 491910/500159 [00:06<00:00, 79352.41KB/s]
    100%|#########9| 499857/500159 [00:06<00:00, 78741.45KB/s]
    100%|##########| 500159/500159 [00:06<00:00, 73971.77KB/s]
    Downloading /root/.mxnet/models/vgg16_ucf101-d6dc1bba.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/vgg16_ucf101-d6dc1bba.zip...

      0%|          | 0/486863 [00:00<?, ?KB/s]
      0%|          | 101/486863 [00:00<09:25, 861.43KB/s]
      0%|          | 515/486863 [00:00<03:22, 2398.61KB/s]
      0%|          | 2187/486863 [00:00<01:02, 7790.81KB/s]
      2%|1         | 8017/486863 [00:00<00:18, 25783.45KB/s]
      3%|3         | 14838/486863 [00:00<00:11, 40091.01KB/s]
      4%|4         | 20502/486863 [00:00<00:10, 45470.18KB/s]
      6%|5         | 27947/486863 [00:00<00:08, 52272.19KB/s]
      8%|7         | 36694/486863 [00:00<00:07, 62911.05KB/s]
      9%|9         | 44678/486863 [00:00<00:06, 68014.72KB/s]
     11%|#         | 52303/486863 [00:01<00:06, 70489.27KB/s]
     12%|#2        | 60337/486863 [00:01<00:05, 73450.34KB/s]
     14%|#4        | 68779/486863 [00:01<00:05, 76744.97KB/s]
     16%|#5        | 76496/486863 [00:01<00:05, 74867.84KB/s]
     17%|#7        | 84022/486863 [00:01<00:05, 74726.04KB/s]
     19%|#8        | 92041/486863 [00:01<00:05, 76338.98KB/s]
     21%|##        | 100279/486863 [00:01<00:04, 78131.49KB/s]
     22%|##2       | 108750/486863 [00:01<00:04, 80082.47KB/s]
     24%|##4       | 117447/486863 [00:01<00:04, 82129.02KB/s]
     26%|##5       | 125725/486863 [00:01<00:04, 82320.97KB/s]
     28%|##7       | 134062/486863 [00:02<00:04, 82575.26KB/s]
     29%|##9       | 142945/486863 [00:02<00:04, 84438.21KB/s]
     31%|###1      | 151394/486863 [00:02<00:04, 83855.77KB/s]
     33%|###2      | 159784/486863 [00:02<00:03, 83619.15KB/s]
     35%|###4      | 168549/486863 [00:02<00:03, 84817.07KB/s]
     36%|###6      | 177034/486863 [00:02<00:03, 84332.44KB/s]
     38%|###8      | 185470/486863 [00:02<00:03, 83978.42KB/s]
     40%|###9      | 193976/486863 [00:02<00:03, 84298.08KB/s]
     42%|####1     | 202604/486863 [00:02<00:03, 84888.09KB/s]
     43%|####3     | 211095/486863 [00:02<00:03, 84255.18KB/s]
     45%|####5     | 219581/486863 [00:03<00:03, 84412.42KB/s]
     47%|####6     | 228131/486863 [00:03<00:03, 84735.36KB/s]
     49%|####8     | 236606/486863 [00:03<00:02, 84384.64KB/s]
     50%|#####     | 245046/486863 [00:03<00:02, 83365.53KB/s]
     52%|#####2    | 253403/486863 [00:03<00:02, 83424.59KB/s]
     54%|#####3    | 261783/486863 [00:03<00:02, 83535.59KB/s]
     56%|#####5    | 270380/486863 [00:03<00:02, 84079.17KB/s]
     57%|#####7    | 278873/486863 [00:03<00:02, 84331.63KB/s]
     59%|#####9    | 287308/486863 [00:03<00:02, 84046.85KB/s]
     61%|######    | 295781/486863 [00:03<00:02, 84248.99KB/s]
     63%|######2   | 304439/486863 [00:04<00:02, 84943.53KB/s]
     64%|######4   | 312935/486863 [00:04<00:02, 83879.66KB/s]
     66%|######5   | 321327/486863 [00:04<00:01, 83619.83KB/s]
     68%|######7   | 330319/486863 [00:04<00:01, 85476.39KB/s]
     70%|######9   | 338870/486863 [00:04<00:01, 84535.53KB/s]
     71%|#######1  | 347328/486863 [00:04<00:01, 84126.56KB/s]
     73%|#######3  | 355967/486863 [00:04<00:01, 84795.94KB/s]
     75%|#######4  | 364450/486863 [00:04<00:01, 84663.47KB/s]
     77%|#######6  | 372919/486863 [00:04<00:01, 84196.48KB/s]
     78%|#######8  | 381532/486863 [00:05<00:01, 84767.77KB/s]
     80%|########  | 390011/486863 [00:05<00:01, 84564.13KB/s]
     82%|########1 | 398469/486863 [00:05<00:01, 83788.37KB/s]
     84%|########3 | 406850/486863 [00:05<00:00, 82494.52KB/s]
     85%|########5 | 415199/486863 [00:05<00:00, 82784.89KB/s]
     87%|########6 | 423482/486863 [00:05<00:00, 82451.18KB/s]
     89%|########8 | 432047/486863 [00:05<00:00, 82731.13KB/s]
     91%|######### | 440845/486863 [00:05<00:00, 84282.41KB/s]
     92%|#########2| 449338/486863 [00:05<00:00, 84472.04KB/s]
     94%|#########4| 457788/486863 [00:05<00:00, 83029.48KB/s]
     96%|#########5| 466693/486863 [00:06<00:00, 84802.73KB/s]
     98%|#########7| 475181/486863 [00:06<00:00, 82984.17KB/s]
     99%|#########9| 483492/486863 [00:06<00:00, 82702.06KB/s]
    486864KB [00:06, 77608.12KB/s]                            




.. GENERATED FROM PYTHON SOURCE LINES 82-85

Note that if you want to use InceptionV3 series model, please resize the image to have
both dimensions larger than 299 (e.g., 340x450) and change input size from 224 to 299
in the transform function. Finally, we prepare the image and feed it to the model.

.. GENERATED FROM PYTHON SOURCE LINES 86-97

.. code-block:: default


    pred = net(nd.array(img_list[0]).expand_dims(axis=0))

    classes = net.classes
    topK = 5
    ind = nd.topk(pred, k=topK)[0].astype('int')
    print('The input video frame is classified to be')
    for i in range(topK):
        print('\t[%s], with probability %.3f.'%
              (classes[ind[i].asscalar()], nd.softmax(pred)[0][ind[i]].asscalar()))





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    The input video frame is classified to be
            [ThrowDiscus], with probability 0.998.
            [HorseRace], with probability 0.001.
            [VolleyballSpiking], with probability 0.001.
            [Hammering], with probability 0.000.
            [TennisSwing], with probability 0.000.




.. GENERATED FROM PYTHON SOURCE LINES 98-100

We can see that our pre-trained model predicts this video frame
to be ``throw discus`` action with high confidence.

.. GENERATED FROM PYTHON SOURCE LINES 103-105

The next example is how to perform video action recognition, e.g., use the same
pre-trained model on an entire video.

.. GENERATED FROM PYTHON SOURCE LINES 108-117

First, we download the video and sample the video frames at a speed of 1 frame per second.

.. raw:: html

    <div align="center">
        <img src="../../_static/action_basketball_demo.gif">
    </div>

    <br>

.. GENERATED FROM PYTHON SOURCE LINES 118-138

.. code-block:: default


    from gluoncv.utils import try_import_cv2
    cv2 = try_import_cv2()

    url = 'https://github.com/bryanyzhu/tiny-ucf101/raw/master/v_Basketball_g01_c01.avi'
    video_fname = utils.download(url)

    cap = cv2.VideoCapture(video_fname)
    cnt = 0
    video_frames = []
    while(cap.isOpened()):
        ret, frame = cap.read()
        cnt += 1
        if ret and cnt % 25 == 0:
            video_frames.append(frame)
        if not ret: break

    cap.release()
    print('We evenly extract %d frames from the video %s.' % (len(video_frames), video_fname))





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none

    Downloading v_Basketball_g01_c01.avi from https://github.com/bryanyzhu/tiny-ucf101/raw/master/v_Basketball_g01_c01.avi...

      0%|          | 0/281 [00:00<?, ?KB/s]
    282KB [00:00, 59973.32KB/s]            
    We evenly extract 0 frames from the video v_Basketball_g01_c01.avi.




.. GENERATED FROM PYTHON SOURCE LINES 139-141

Now we transform each video frame and feed them into the model.
In the end, we average the predictions from multiple video frames to get a reasonable prediction.

.. GENERATED FROM PYTHON SOURCE LINES 142-158

.. code-block:: default

    if video_frames:
        video_frames_transformed = transform_fn(video_frames)
        final_pred = 0
        for _, frame_img in enumerate(video_frames_transformed):
            pred = net(nd.array(frame_img).expand_dims(axis=0))
            final_pred += pred
        final_pred /= len(video_frames)

        classes = net.classes
        topK = 5
        ind = nd.topk(final_pred, k=topK)[0].astype('int')
        print('The input video is classified to be')
        for i in range(topK):
            print('\t[%s], with probability %.3f.'%
                  (classes[ind[i].asscalar()], nd.softmax(final_pred)[0][ind[i]].asscalar()))








.. GENERATED FROM PYTHON SOURCE LINES 159-162

We can see that our pre-trained model predicts this video
to be ``basketball`` action with high confidence.
Note that, there are many ways to sample video frames and obtain a final video-level prediction.

.. GENERATED FROM PYTHON SOURCE LINES 167-172

Next Step
---------

If you would like to dive deeper into training TSN models on ``UCF101``,
feel free to read the next `tutorial on UCF101 <dive_deep_tsn_ucf101.html>`__.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  30.899 seconds)


.. _sphx_glr_download_build_examples_action_recognition_demo_tsn_ucf101.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example



  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: demo_tsn_ucf101.py <demo_tsn_ucf101.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: demo_tsn_ucf101.ipynb <demo_tsn_ucf101.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_