machine learning | Yifei Li

A Method for Automatically Animating Children’s Drawings of the Human Figure

Wed, 01 Feb 2023 00:00:00 +0000

Code 💻

Follow our Github repository Animated Drawings

Demo

Try animating your own drawing Here!

Technical Posts

Dataset of 180,000 annotated drawings: A new, unique AI dataset for animating amateur drawings
Meta AI Technical post: Using AI to bring children’s drawings to life
Meta AI interview: Meet the researcher using AI to build children’s animations

Dataset

We have open sourced our dataset of ~180,000 annotated drawings, which you can explore for your own research.

Website

Go to here

Video

Citation

@article{10.1145/3592788,
author = {Smith, Harrison Jesse and Zheng, Qingyuan and Li, Yifei and Jain, Somya and Hodgins, Jessica K.},
title = {A Method for Animating Children’s Drawings of the Human Figure},
year = {2023},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
issn = {0730-0301},
url = {https://doi.org/10.1145/3592788},
doi = {10.1145/3592788},
abstract = {Children’s drawings have a wonderful inventiveness, creativity, and variety to them. We present a system that automatically animates children’s drawings of the human figure, is robust to the variance inherent in these depictions, and is simple and straightforward enough for anyone to use. We demonstrate the value and broad appeal of our approach by building and releasing the Animated Drawings Demo, a freely available public website that has been used by millions of people around the world. We present a set of experiments exploring the amount of training data needed for fine-tuning, as well as a perceptual study demonstrating the appeal of a novel twisted perspective retargeting technique. Finally, we introduce the Amateur Drawings Dataset, a first-of-its-kind annotated dataset, collected via the public demo, containing over 178,000 amateur drawings and corresponding user-accepted character bounding boxes, segmentation masks, and joint location annotations.},
note = {Just Accepted},
journal = {ACM Trans. Graph.},
month = {apr}
}

Acknowledgement

We would like to thank FAIR Interfaces, FAIR X, and other members of Meta who helped in the building and release of the demo

Keywords

graphics, animation, image manipulation, machine learning, drawings, children drawings, non-photorealistic rendering, line drawing, computer graphics, dataset, image dataset

DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact

Fri, 01 Apr 2022 00:00:00 +0000

Keywords

diffcloth, projective dynamics, jacobi solver, cloth simulation, differentiable simulation, physics simulation, inverse problems

Code 💻

Follow our Github repository Diffcloth

Demos

For details refer to the paper Sec 6. Applications.

Hat: Trajectory Optimization

Optimize manipulator end effector trajectories to move the hat onto the head.
1737 Dof, h=1/100s, 400 Timesteps, 18 Design Parameters

Sock: Trajectory Optimization

Optimize manipulator end effector trajectories to put on the sock.
3165 Dof, h=1/160s, 400 Timesteps, 36 Design Parameters

Hat-Controller: Closed-Loop Control

We train a generalizable closed-loop controller that can put on the hat from different initial positions.
1737 Dof, h=1/100s, 400 Timesteps, 117126 Design Parameters

Dress: Inverse Design

Optimize dress material parameters so that the spinning angle of the dress is 50 degrees.
10902 Dof, h=1/120s, 125 Timesteps, 2 Design Parameters

Sphere: System Identification

Optimize the fricitonal coefficient between the sphere and the cloth to match target trajectory.
1875 Dof, h=1/180s, 350 Timesteps, 1 Design Parameters

T-shirt: System Identification

Optimize wind model and cloth material parameters to match target trajectory.
4278 Dof, h=1/90s, 250 Timesteps, 6 Design Parameters

Demo

Presentation

Citation

@article{li2022diffcloth,
author = {Li, Yifei and Du, Tao and Wu, Kui and Xu, Jie and Matusik, Wojciech},
title = {DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact},
year = {2022},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
issn = {0730-0301},
url = {https://doi.org/10.1145/3527660},
doi = {10.1145/3527660},
abstract = {Cloth simulation has wide applications in computer animation, garment design, and robot-assisted dressing. This work presents a differentiable cloth simulator whose additional gradient information facilitates cloth-related applications. Our differentiable simulator extends a state-of-the-art cloth simulator based on Projective Dynamics (PD) and with dry frictional contact&nbsp;[Ly et&nbsp;al. 2020]},
note = {Just Accepted},
journal = {ACM Trans. Graph.},
month = {mar},
keywords = {cloth simulation, differentiable simulation, Projective Dynamics}
}

Acknowledgement

We thank Marco Renedo for his helpful discussions on the preconditioners, Junbang Liang for his help with running the baseline comparison code, and the anonymous reviewers for their helpful comments. This work was supported in part by the Defense Advanced Research Projects Agency (DARPA) under grant No. FA8750-20-C-0075.

JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints

Tue, 01 Mar 2022 00:00:00 +0000

Code 💻

Follow our Github repository JoinABLe

Motivation

Physical products are often complex assemblies combining a multitude of 3D parts modeled in computer-aided design (CAD) software. CAD designers build up these assemblies by aligning individual parts to one another using constraints called joints. JoinABLe is a learning-based method that assembles parts together to form joints. JoinABLe uses the weak supervision available in standard parametric CAD files without the help of object class labels or human guidance. Our results show that by making network predictions over a graph representation of solid models we can outperform multiple baseline methods with an accuracy (79.53%) that approaches human performance (80%).

Task

Given a pair of parts, we aim to create a parametric joint between them, such that the two parts are constrained relative to one another with the same joint axis and pose as defined by the ground truth.

Representation

JoinABLe takes as input a pair of parts in the B-Rep format. We represent each part as a graph derived from the B-Rep topology. Graph vertices are either B-Rep faces or edges and graph edges are defined by adjacency.

Architecture

Our overall architecture is shown in the figure below and consists of an encoder module that outputs per-vertex embeddings for each B-Rep face and edge in our graph representation of the input parts. Using these embeddings we can predict a joint axis and then search for joint pose parameters.

Citation

 @inproceedings{willis2022joinable,
title = {JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints},
author = {Willis, Karl DD and Jayaraman, Pradeep Kumar and Chu, Hang and Tian, Yunsheng and Li, Yifei and Grandi, Daniele and Sanghi, Aditya and Tran, Linh and Lambourne, Joseph G and Solar-Lezama, Armando and Matusik, Wojciech},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022}
}