With the hackathon coming up I thought we could ponder what could be done
while there. I've been constructing a list of horrible ideas over the last
couple weeks:
Web UI for cirrus debug/devel features:
- Settings dump
- Mappings dump
- Copy version of settings+mappings suitable to create index with curl
- cirrusDumpQuery
- cirrusDumpResult
- cirrusExplain
- cirrusUserTesting
Top level idea is to make it easy to access all of these things. Could be
a userscript run on-page in the wiki. Could be an SPA run from tool labs
(or even
people.wikimedia.org).
============
docker setup to initialize elasticsearch, import latest cirrus dump, and
attach a kibana instance for UI. Probably with a modified mapping more
amicable to kibana inspection.
============
Some script to manage elasticsearch allocation manually via api? Pointless,
but
perhaps fun.
===========
phabricator formatted export for jupyter
- problem: images?
-- Seems would need to upload separately and then reference them in final
output
-- There is an api for this, but then we can't just emit something to paste
into a field
the whole export needs to happen over api then.
- better, but worse: data-uri's would be great. But i dunno if phab is
built for megabyte sized posts. They also
don't support data-uri's. Browsers also hate when you copy/paste
excessive amounts of data.
==========
Custom implementation to find similar images in commons:
-
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.673.5151&rep=r…
-
http://www.deepideas.net/building-content-based-search-engine-quantifying-s…
- Convert image into a feature vector
- Use clustering to generate an image signature
- Find k-nearest-neighbors via Earth Mover Distance (EMD), can utilize
pyemd library.
- It's very not-obvious how the signature + weight gets plugged into pyemd
- EMD is expensive, no clue how this would scale to millions of images
- This would probably perform poorly, more interesting to get to understand
some of the history of similar image retrieval
=========
https://github.com/beniz/deepdetect.git ?
- Use pre-trained ML to detect objects in images and then label those
objects.
- Can compare similarity of objects detected for similar images. Can
probably
extend with color information
- Do we actually have a use case for images similar to other images?
Perhaps on upload?
==========
Elasticsearch cluster balance simulator
- Allow to Simulate valuate how the cluster balancing performs under
various simulated conditions
- no way this could be done in a weekend hackathon. It would probably be
completely wrong as well and simulate some idealized cluster that doesn't
act
like ours.
==========
Prototype Lire plugin for elasticsearch
- Lire = Lucene Image REtrieval
- I know nothing about it, other than it exists
- Plugin already exists plugging it into solr, so how hard could it be?
- Maybe try it out standalone with some small test set to see what it does