Video Mosaic Component Programs

The video mosaic process has been broken up into four programs.  The primary reason for this is so that operations that require movie compression/decompression/manipulation are separated from the actual mosaic search itself.  That way, the mosaic search can (and has) be written for multiplatform compilation and can be run on the fastest machine available.

• The width of the tiles.
• The height of the tiles.
• The number of bins to use in metrics that involve histograms.
• The location of where to save tile database entries.
• The location of the actual video clips to process.
• The video clip to start on (so database generation can be parallelized).
• The number of video clips to process (also for parallelization).

In addition, the following pertinent compiler predefinition have been placed in an interface file to the utilities file that all programs use:

• The factor of tile size by which to resize frames for metrics that require smaller frames. 
• Various metric-related constants, such as thresholds.
• The number of frames per tile.

VideoDBGen works with video via the QuickTime SDK, which can handle movies of nearly any compression.  Each movie in the video repository is read in and decompressed.  The movie is then divided into tiles of width and height indicated by the user. The number of frames in a tile is indicated by a compiler predefinition in a shared library.

A database consists of a folder with file entries for each clip.  For each tile of each movie, metrics are run and a feature vector generated.  These vectors are written to the database entry file.  Tile metadata (i.e. frame location, tile number, etc) is associated with each vector.  Movie metadata is written at the top of each database entry file.  Presently, the database entries are written out in ASCII text for debug reasons, although a library is already in place to do binary I/O with these files.

VideoMosaicPrep
This program takes the following command-line parameters:

• The width of the tiles.
• The height of the tiles.
• The number of bins to use in metrics that involve histograms.
• The duration of each tile in frames.
• The amount of frames dedicated to crossfade between tiles in the final mosaic.
• The input file location and name.
• A work directory for any temporary files that need to be created.

VideoMosaicPrep also uses the QuickTime SDK.  A movie is read in and decompressed.  The movie is then divided into tiles of width and height indicated by the user. The number of frames in a tile is randomized slightly so that all the tiles don't start and end at the same time.

Once tiles have been created, this program then calculates the optical flow between each frame in the entire movie, and then writes this information to the output file.  Next, it iterates through them and generates feature vectors, which are then written to a file.  Tile metadata (i.e. frame location, tile number, etc) is associated with each vector.

VideoMosaic
This program takes the following command-line parameters:

• The output file of the MosaicMakePrep program.
• The location of all the tile database.
• A stat file with pre-calculated metric normalization data, if any.
• A stat file with pre-calculated statistics on tile comparisons, if any.
• The absolute filename of the mosaic output file.

The VideoMosaic program does the meat of the mosaic creation process: the actual search for tile replacements.  The program is very CPU and memory intensive, and does not make use of any OS-dependant SDKs so that it could be ported easily to faster *nix machines.

This program takes in a video mosaic prep file and then searches the database for the best fit tile for each tile in the original video.  The search is linear and has been optimized to 3 tight loops in C++. 

In addition, a number of statistical files can be read in to modify how vectors are compared.  Because the values of the features are distributed very differently, one of the tactics used is to attempt to normalize them based on statistics of samples of comparison values. This is so that some features can be proportionally weighted more than others.

MosaicMakeVideo
This program takes the following command-line parameters:

• The output file from the VideoMosaic program containing a record of original and replacement tiles. 
• The location of the video clip repository.
• A work directory for any temporary files that need to be created.

This program uses the QuickTime SDK to build actual mosaic movies from the results of the VideoMosaic search.  The input movie is decompressed in memory, and then tiles of it are replaced with tiles from other movies in the repository.  When a tile is replaced it is faded into its replacement to provide a more seamless effect.

This program is I/O bound and has been optimized so that creating a mosaic video is pretty efficient.

Various Statistical Calculation Programs
To do quick post-op calculations of statistics on database files, several perl scripts were written.  The more useful of these include:

• DiffStats - Calculates an estimate of the mean and standard deviation of feature vector comparison results.  The estimate is generated by random sampling, both of database entries and of feature vectors within those entries.  These statistics are then spat out to a file which can be used to normalize feature values for comparison in VideoMosaic.
• WriteMax - Calculates the max, min, mean, and standard deviation of all the feature values in the database given it.  These are written out to files.
• GenMetricsHist - Generate histograms of metric comparison data for easy analysis of feature contribution.

Sections

1. Section 1: Video Mosaic Creation Process
2. Section 2: Video Mosaic Component Programs
3. Section 3: Video Mosaic Features
4. Section 4: Some Video Mosaic Results
5. Section 5: Work In-Progress and Future Goals