Tu Bui¹, Leonardo Ribeiro², Moacir Ponti², John Collomosse¹

¹Centre for Vision, Speech and Signal Processing

University of Surrey, UK.

²Institute of Mathematics and Computer Sciences

University of Sao Paulo, Brazil.

We propose and evaluate several deep network architectures for measuring the similarity between sketches and photographs, within the context of the sketch based image retrieval (SBIR) task. We study the ability of our networks to generalize across diverse object categories from limited training data, and explore in detail strategies for weight sharing, pre-processing, data augmentation and dimensionality reduction. In addition to a detailed comparative study of network configurations, we contribute by describing a hybrid multi-stage training network that exploits both contrastive and triplet networks to exceed state of the art performance on several SBIR benchmarks by a significant margin.

Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression (pdf).

1. Dataset

TU-Berlin Sketch (training): 20,000 sketches of 250 categories obtained from Amazon Mechanical Turk (Source).

Flickr25K (training): 25,000 images of the same 250 categories, resized to max dimension of 256 pixels, crawled from Flickr, Google and Bing (1.8G) (Download).

Flickr15K (test): ≈15,000 images and 330 sketches of 33 categories (Source, Mirror).


2. Pretrained model

Caffe pretrained model (90MB) (Download)


3. Demo




4. Code

Python code for feature extraction and deploy prototxt for the pretrained model (Github)


5. BibTeX

@article{bui2018sketching,
  title={Sketching out the Details: Sketch-based Image Retrieval using Convolutional Neural Networks with Multi-stage Regression},
  author={Bui, Tu and Ribeiro, Leonardo and Ponti, Moacir and Collomosse, John},
  journal={Computers \& Graphics},
  year={2018},
  volume={71},
  pages={77--87},
  publisher={Elsevier}
}