Video Mosaic Creation Process

To make a video mosaic, the following steps are completed:

  1. Create the Tile Database.  There are 160GB of stock video available to be used in creating a mosaic.  I currently use  272 movies at a 720x480 resolution averaging 200 frames apiece.  This is about 10% of the movies I have available.

    To create the tile database, the VideoDBGen program loads each movie into memory, decompresses it, and divides it into tiles in both time and frame size.  The number of frames per tile is a compiler definition, and the spatial dimensions are specified on the command line. The feature generation metrics described earlier are then run on each tile, and the resulting vector is written to a database file.

    The completed tile database consists of a folder with one file per movie in the database.  With 30x30 tiles, the database is about 1GB in size.


  2. Prepare the Target Movie.  Once a movie is selected to have a mosaic made, the VideoMosaicPrep program is run on it.  This program loads the video into memory, decompresses it, and divides it into tiles in both time and frame size.  The per-frame optical flow is then calculated and recorded to a movie description file.  Finally, the feature generation metrics described earlier are then run on each tile, and the resulting vector is also saved.


  3. Search the Tile Database.  Using the VideoMosaic program, search the database for the best match for each video tile in the original video.  This is basically a linear search using direct comparisons of feature vectors.  Despite being such a direct approach, this operation has been written in C and optimized such that a typical mosaic run completes in about 20 minutes.  The output is a file containing the best match for each tile and where to find it, along with comparison data for metric evaluation.


  4. Create the Actual Mosaic.  MakeMosaicMovie is run after creating a mosaic description file.  The original movie is loaded and decompressed in memory, and then each tile of it is replaced with tiles from the video repository.  The tiles are blended together with slight regions of overlap using a linear blend.  This program has also been optimized, so the actual mosaic movie creation process is on the order of 10-20 minutes.  Uncompressed frames are output to a directory on completion.

 

 

Sections

  1. Section 1: Video Mosaic Creation Process
  2. Section 2: Video Mosaic Component Programs
  3. Section 3: Video Mosaic Features
  4. Section 4: Some Video Mosaic Results
  5. Section 5: Work In-Progress and Future Goals