WNImage

Todo

• Homogeneity penalty.

Website

A website with the current results can be found at http://psy-build2.princeton.edu/wnimage. There are results for 1000 images using a random seed of 8675309.

The files are named top5synsets_alpha_gamma_rho.html. The parameters have the following meanings:

alpha

Determines how the score for a synset within an Xsynset decays as a function of the path length from the generator synset to the given synset.

gamma

Words which are potentially colors are weighted by this factor.

rho

Determines how the score for a synset within an Xsynset decays as a function of its shallowness from the root of the wordnet database.

These results are then used to assign images to synsets (illustrate synsets). These assignments are in files named assignments_alpha_gamma_rho.html (similar to above).

CVS Access

The WNImage tools in the repository under wnimage. The repository is named wnp. To access it, follow the instructions at [1].

Files

CVS Files

• gen_Xsynsets.py - generates the Xsynset database file for a given list of words on stdin. The db gets pickled to Xsynsetdb.pkl. The max_depth parameter specifies how many links to follow. It currently only follows hypernyms and it crawls all senses of a word.
• gen_weighted_Xsynsets.py - generates the wXsynset database file all the Xsynsets in Xsynsetdb.pkl and outputs the new pickled db as wXsynsetdb.pkl. Using alpha=0.5, it computed a weighted Xsynset, i.e. one which simply has a numerical value assigned to each word. It is currently alpha^path_length from source to target.
• gen_caption_vects.py - generates a db of the weighted (alpha=0.5) Xsynset for each image caption. pass in the captions file as the first parameter. You need to run this after gen_Xsynsets.py and it expects the result of that to be named Xsynsetdb.py. Note: this generates a pretty large file and it doesn't save much time so it might be scrapped.
• rank_caption_synsets.py - ranks the top N (second parameter) synsets for all the captions. The captions file is the first parameter.
• assign_images.py - assigns images to synsets. Takes in a results file (output of rank_caption_synsets.py) on stdin, and prints out the assignments on stdout. Currently just picks the top one.

• extract_caption_words.sh - extracts and uniquifies all the words in the captions of a captions file. Input on stdin and output on stdout.
• Xsynsettools.py - a library for utility functions relating to Xsynsets. Currently just has a function to generate Xsynsets.
• similarity.py - a library for similarity computations. Currently just has cosine similarity.
• captionstools.py - a library for utility functions relating to image caption manipulation (i.e. reading, vector extraction, etc.)
• cluster.py - a library with clustering routines. It currently just implements k-means.
• chi2.py - a library for computing chi^2 statistics.
• results2html.sh - takes the output of rank_caption_synsets.py and generates a decent looking webpage.
• assignments2html.sh - takes the output of assign_images.py and generates a decent looking webpage.

Experimental Files

These are large data files (too large and/or time consuming to put in everybody's CVS). All files are in on psy-build2 at /wnimage.

• captions - This is the captions file that has been dos2unix-ified.
• captionwords - A sorted and uniquified list of all the words that occur in the captions file.
• Xsynsetdb.pkl - This is the database of Xsynsets generated using gen_Xsynsets.py. You should create a simlink from this file to your working dir.
• wXsynsetdb.pkl - This is the database of weighted Xsynsets generated using gen_weighted_Xsynsets.py. You should create a simlink from this file to your working dir.

The results subdirectory contains working results for experiments.

• top5synsets_alpha_gamma_rho - The top 5 synsets for each image caption. Raw results named analogously to top5synsets_alpha_gamma_rho.html (see above). These should be fed into results2html.sh or assign_images.py.
• assignments_alpha_gamma_rho - The raw results fed into assignments2html.sh.

Notes

5/9/06

• The goal of the project is to use Xsynsets to associate images with synsets (read: illustrate synsets).
• The first step of this is to do the opposite, namely to rank Xsynsets for each image.
  • Once we have a scored ranking, we can adjust competition parameters to determine how to assign it to a synset.
    • Homogeneity penalty which devaluates the assignment if the image could be associated to many synsets (because there are many synsets of a similar high-score).
  • We will also explore prior bias parameters.
    • Lower the weight given to colors.
    • Lower the weight the further up the hypernym hierarchy a word is (in principle, giving more weight to more specific terms).
  • Last, we can explore different ways of computing and using extended synsets.
    • Change the function between path length and score (a^length vs. alpha/(alpha+length)).
    • Weight different relations differently.
    • Boosting words that are synset kings.
    • Normalization via out degree.
• I will set up a large (~1000) set of images on the website using a randomized seed with the results of different approaches so that we can judge each of the techniques.
• In order to test our technique eventually, we will give users a synset (+gloss) and ask them to judge how representative the picture is of the synset (summer intern).
• XM will implement basic similarity between captions and synset glosses as a baseline technique.

5/5/06

• The principal goal (or first milestone) of this project is to use Xsynsets rank the synsets associated with the given image.
• Each Xsynset will be implemented in python as a dictionary. In summary, an Xsynset uses the following structures:

  • synset 

    the synset number within wordnet is recorded.

    path 

    a path is a list of synsets (starting node to ending node).

    typed_path 

    a tuple where the first element is a wordnet connection type and the second is a path. This represents a path through wordnet where all the traversed edges are of the type specified in the first element of the tuple.

    entry 

    is a dictionary entry where the key is a synset and the value is a list of typed_paths. The list of typed_paths are all those paths which go from the Xsynset's generator synset to the given target synset, while only traversing one type of connection.

    Xsynset 

    is a list of entries. If a synset does not appear as any key in the Xsynset, then it cannot be reached from the generator synset within the threshold number of steps.

5/3/06

JBG has done some basic disambiguation using Lesk. There are pages for both synsets and each individual image. The results are pretty shoddy.

Personal Work Notes

• Jonathan's Lab Notes