Symmetry Analogies

From CSWiki
Jump to: navigation, search

This is a discussion page for symmetry-space editing framework. People currently involved in this project are: Vladimir Kim, Yaron Lipman and Thomas Funkhouser. Feel free to contact us if you would like to participate. Some examples can be found here.

Examples Taxonomy

Classification Legend

Types of input/target

  • Input f Deformable image
    • f1 One smooth image s.t. all of its parts are smoothly related via some transformation. E.g. face, human pose,
    • f2 Disconnected components related by some transformation. E.g. windows, leafs, cars in the traffic
  • Target g Target symmetry image
    • g1 Real image. E.g. same as f1, but with different subject.
    • g2 Synthetic (perfect) shape. E.g. Perfect oval, plane of symmetry
    • g3 None. E.g. Target is defined via symmetry maximization/minimization.

D Types of deformations

  • D1 Surface control points. E.g. piecewise linear coords, human pose angles, patch coordinates (cut patches from some image f, blend them in using Efros & Freeman quilting technique.)
  • D2 Warp. E.g. move control points in overlay lattice and interpolate values in-between, caged mesh
  • D3 Semantic Deforms. E.g. move face elements only to create valid human face, restrict human pose angles to have valid human, etc...

E Types of energy

  • E1 ||S(g) - S(f)|| Difference in symmetry space
  • E2 ||S(f)||, if g is not provided, and where ||Sf|| is assumed to specify how symmetric is object with respect to symmetry operator S.

S Types of Symmetry operators. S(f) maps smooth function to its symmetry space.

  • S={pointwise, translation, PRST}, and in addition:
    • S_masked masked based on area of influence (e.g. local vs global)
    • S_thresh masked based on strongest response (e.g. plane of symmetry)
    • S_norm normalized (e.g. in the beginning or at each step)

Valid combination table

Note that:

  • Input image (f) + Deformation (D) => Space of valid solutions
  • Energy (E) + target image (g) => Minimal energy solution (i.e. output).

So, space of valid solutions can be described as:

D1: Surface Ctrl Pnts D2: Warp D3: Semantic-aware
f1: Single Shape 1. Human in all possible poses (vary angles of skeleton)
2. All surfaces/curves with given topology (vary coordinates of vertices)
3.Image with hole filled with some patch locations (vary coordinates of patches)
Object Image/Mesh under non-rigid transformations (vary cage control points) Human Faces with some elements moved (vary location and dimensions of face elements)
f2: Many Shapes 1. Texture consisting of quilts (vary location of quilts)
2.Image with hole filled
Objects Image: Cars/Windows/regular deformed (some parts shrunk/expanded) Leafs on tree moved along branch

So, for each input the desired output depends on Energy + target symmetry S(g) combination.

E=L2[S(g_real) - S(f)] E=L2[S(g_synth) - S(f)] E=Energy[S(f)]
f1D1 1. Canonical Pose Align human to other human's pose.
2. Symmetry Transfer of surface
1. Canonical Pose Align human to sketch pose (or perfect pose)
2. Symmetry-by-sketch transfer symmetry of user-specified sketch
1. Symmetrization Make human pose perfectly symmetric (with respect to plane, translations, etc...)
2. Symmetrization Make shape perfectly symmetric (keeping topology)
f1D2 Symmetry TransferTransfer symmetry of another image/surface. Symemtry-by-sketch, Symmetrization, Assymetrization Transfer symmetry of user sketch. Transfer symmetry of (im)perfect shape.. Symmetrization Make surface/image perfectly symmetric.
f1D3 Symmetry Transfer Copy face symmetries of one human to another. Note: only source needs user input control points. Symmetry Transfer, Assymm, Symm, Symm-by-sketch Make face symmetric/assymetric as in given synthetic example. Change plane of symmetry of a given face. Symmetrization Make face more symmetric. Note: changing S should produce interesting results, should face have more planar symmetry? more translation symmetry?
f2D1 1. Texture Transfer, Symmetry Transfer Give texture symmetries of some other real texture (e.g. translational pattern of tree cut of tree), translational pattern of leaves.
2. Inpainting Fill hole with some patches to have same symmetry as in another image example
1. Texture Transfer Have texture with 'as perfect as possible' pattern: checkerboard, mirror reflection, given imperfect patches.
2. Inpainting Fill hole so that image is 'as perfect as possible'
1. Texture Synthesis Have texture that is as symmetric as possible
2. InpaintingFill hole such that image is as symmetric as possible.
f2D2 Symmetry Transfer Try to align objects as in another image Symmetrization/Assymetrization/Symmetry Transfer Align objects in a perfect pattern Symmetrization Align objects to have perfect symmetry
f2D3 Symmetry Transfer Align leaves as in another tree Symmetrization/Assymetrization Align leaves in a perfect pattern Symmetrization Align leaves perfectly with respect to symmetry.


Example Details

  • Inpainting f=image with hole, copy patches from f on itself to maximize symmetry. E2, D3, any S_norm.
  • Texture Synthesis Copy quilts s.t. either target symmetries of g are achieved or symmetry is maximized. Any E, D1, any S. Note: varying S should result in different images, especially if g is not given.
  • Symmetry Transfer Copy symmetry of one object (possibly synthetic) to another object.
    • Copy symmetry of faces - might require semantic details, and putting control points
    • Copy symmetry of humans to align them into same 'canonical' pose. One approach: extract skeleton from target pose (mesh g) and source pose (mesh f). Find Sf and Sg, and optimize with respect to E1.
    • Align shape with respect to plane of symmetry - e.g. user draws line/plane of symmetry and shape becomes more symmetric with respect to this line
  • Symmetrization Make shape more symmetric. Note that this can be also achieved via symmetry transfer, where target is some symmetric synthetic image/shape. For any deformation model optimize with respect to E2.
    • Mesh symmetrization
  • Assymetrization make some shape less symmetric. f=regular image, g=asymmetric image.
    • Pseudo-random points distribution f=regular points, S(g) = S(f) + gaussian blur.

To do

  • Inpainting
  • Texture Synthesis
  • Canonical Pose in 3D
    • Finding skeleton
    • PRST in 3D

Code

The most recent MATLAB code can be obtained from Vladimir. This section is just a way to keep track of what has been implemented.

Framework

  • ./transferSymmetry.m - transfers symmetry of g to symmetry of f, via minimizing E=||S(g)-S(f)||
  • ./transferSymmetryEnergy.m - function used by transferSymmetry.m script, calculates E for given S.
  • ./symmetryEnergy.m - attempts to capture how symmetric is the image with respect to given symmetry.
  • ./sampleBilinear.m - samples function f using bilinear interpolation
  • ./synthesizeTexture.m - generates a texture by copying patches (and increasing symmetry energy). Note this method does not need smooth gradients, it is pretty ad-hoc and stochastic at the moment.


Deformation Models

  • Warp
    • ./DefLinear.m - deforms image by moving control points on a regular grid overlay on target image. Triangles are then rasterized onto a target image.
    • ./DefQuadLin.m - deforms image by moving control points overlay on source image. Uniformly-spaced quads are then rasterized onto a target image.
    • ./DeformBSpline.m - similar to above deformations (but with larger support). Not done yet!
    • ./InitDefConst.m - initializes deformation to constant
  • Curves/Skeletons
    • ./convert2DSkeletonToSimple.m - converts skeleton from set of angles to set of absolute coordinates for joints.
    • ./rasterizePieceLinGED.m - generates 2D GED function (e^-dist) distance to line segments.
    • ./rasterizePieceLin.m - generates 2D rendering of a surface
    • ./3D/drawSmoothSlices.m - generates 3D GED function (e^-dist) distance to line segments.
    • ./3D/skeletonMesh3D.m - generates 3D mesh from skeleton
    • ./3D/offsetAdv.m - generates coordinates for lines in fixed-length model (to render & calculating GED).
    • ./3D/skeletonDistVolume.m - calculates distance function from voxel to nearest surface (of piecewise linear skeleton).
  • Texture Synthesis
    • ./DeformCopyPatch.m - copies patches from source image to target canvas
    • ./blendPatchesDM.m - merges foreground patch onto background, using min-cut dynamic programming algorithm
    • ./initTextureSynth.m - initializes texture synthesis structure (used by DeformCopyPatch).


Symmetry Space Maps

  • ./SymSpaceIdentity.m - returns itself. Just as a test if snakes can perform better than symmetries, when f is deformed exactly to g.
  • ./SymSpaceTransFFT.m - FFT version of 2D translational symmetry.
  • ./3D/SymSpaceTrans3DFFT.m - FFT version of 3D translational symmetry.
  • ./SymSpacePntFFT.m - FFT version of 2D pointwise symmetry.
  • ./3D/SymSpacePnt3DFFT.m - FFT version of 3D pointwise symmetry (self-convolution in 3D).
  • ./SymSpacePRST.m - slow brute-force version of PRST, uses code from ./PRST. TODO: implement fft version: SymSpacePRST2DFFT.m
  • ./SymSpaceTransBand.m - Translational symmetry with different levels of locality. TODO: update to fft trans sym.
  • ./getSymmetryMask.m - detect a binary mask (based on thresholding) of symmetry space
  • ./SymMapWithMask.m - only values in masked area actually matter, any symmetry can be used
  • ./SymSpaceNorm.m - normalize symmetry space by constant
  • ./SymSpaceNorm2.m - normalize symmetry space by function value
  • ./testFFTSymmetry.m - tests whether FFT implementation is same as brute-force implementations

Output

  • ./3D/printMeshResults.m - performs optimization (symmetry transfer), generates meshes
  • ./3D/renderMeshes.m - renders meshes pre-computed by printMeshResults.m (use it if machine had no openGL to render meshes at time of calculation).
  • ./printResults.m - prints intermediate images and final optimal image for symmetry transfer.
  • ./plot1DEnergy.m - plots energy E1 or E2 for different values of 1 parameter
  • ./plot2DEnergy.m - plots difference E=|Sf-Sg| energy for different values of 2 parameters

Experiments

NOTE: not all experiments are listed here.

  • Initialization
    • ./initAdvancedHuman.m - initializes human skeleton (fixed-length model)
    • ./3D/initSkeleton3D.m - initializes human skeleton (varying-length model)
    • ./3D/initSkeleton3DAdv.m - initializes human skeleton (fixed-length model)
    • ./Experiments/initCurveHuman.m - initialize human skeleton (varying-length model)
    • ./Experiments/createExample1.m - initialize image with lines (regularly translated & jiggled)
    • ./Experiments/createExample2.m - initialize checkerboard (deformed)
    • ./Experiments/initCurveCircle.m - initialize circle curve
    • ./Experiments/initCurveClosed.m - initialize arbitrary closed curve
    • ./Experiments/initCurveLines.m - initialize line curves (with positions as variables)
    • ./Experiments/initCurveWithJump.m - initialize line curves (with positions as variables), there is an irregular big jump in the middle
  • Experiment scripts