This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
http://wikitweets.herokuapp.com/
wikitweets is a NodeJS [1] application that listens to the Twitter Streaming API [2] for tweets that contain Wikipedia URLs, and then looks up the relevant Wikipedia article using the API to ultimately stream the information to the browser using SocketIO [3]. The most amazing thing for me is seeing the application run comfortably (so far) as a single process on Heroku with no attached database needed. If you are curious the code is on GitHub [4].
The key to wikistream working at all is that Twitter allows you to search and filter the stream using the original (unshorted) URL. So for example a Tweet with the text:
Question of the Day: What’s the greatest seafaring movie ever? Some suggestions: http://bit.ly/IqsE1e (But anything on water'll work) #QOD [5]
Is discoverable with a search query like:
Question of the Day wikipedia.org [6]
Note "wikipedia.org" doesn't exist in the text of the original tweet at all, since it has been shortened by bit.ly -- but it is still searchable because Twitter appear to be unshortening and indexing URLs. Anyhow, I thought I'd share here since this also relied heavily on the various language Wikipedia APIs.
//Ed
[1] http://nodejs.org [2] https://dev.twitter.com/docs/streaming-api/methods [3] http://socket.io [4] https://github.com/edsu/wikitweets [5] https://twitter.com/#!/EWeitzman/status/195520487357558784 [6] https://twitter.com/#!/search/realtime/Question%20of%20the%20Day%20wikipedia...
On Thu, Apr 26, 2012 at 8:02 AM, Ed Summers ehs@pobox.com wrote:
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
That's pretty nicely done and a great snapshot of the collective Wikipedia-consciousness - thanks for sharing!
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
My appreciation too. and the same question, do you also store the records?
bests, .t
On Thu, Apr 26, 2012 at 7:14 PM, emijrp emijrp@gmail.com wrote:
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
-- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT http://code.google.com/p/avbot/ | StatMediaWikihttp://statmediawiki.forja.rediris.es | WikiEvidens http://code.google.com/p/wikievidens/ | WikiPapershttp://wikipapers.referata.com | WikiTeam http://code.google.com/p/wikiteam/ Personal website: https://sites.google.com/site/emijrp/
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Emilio, Taha:
I realize this was long enough ago that you may no longer be interested but I finally got around to adding an archive function to wikitweets [1]. Every time the app collects 1000 tweets that reference Wikipedia it dumps them to a file on Internet Archive [2].
One nice side effect of this is that you get a BitTorrent seed/peer for free [3], which makes mirroring the data pretty simple...if you have a BitTorrent client handy. I blogged a little bit about how it the archive function in wikitweets works [4].
Best, //Ed
[1] http://wikitweets.herokuapp.com [2] http://archive.org/download/wikitweets/wikitweets_archive.torrent [3] http://archive.org/download/wikitweets/wikitweets_archive.torrent [4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/
On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri taha.yaseri@gmail.com wrote:
My appreciation too. and the same question, do you also store the records?
bests, .t
On Thu, Apr 26, 2012 at 7:14 PM, emijrp emijrp@gmail.com wrote:
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
-- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam Personal website: https://sites.google.com/site/emijrp/
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Taha.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Ed,
that's awesome – do you mind adding an entry on the DataHub?
http://thedatahub.org/group/wikimedia
Dario
On Sep 19, 2012, at 6:57 PM, Ed Summers ehs@pobox.com wrote:
Emilio, Taha:
I realize this was long enough ago that you may no longer be interested but I finally got around to adding an archive function to wikitweets [1]. Every time the app collects 1000 tweets that reference Wikipedia it dumps them to a file on Internet Archive [2].
One nice side effect of this is that you get a BitTorrent seed/peer for free [3], which makes mirroring the data pretty simple...if you have a BitTorrent client handy. I blogged a little bit about how it the archive function in wikitweets works [4].
Best, //Ed
[1] http://wikitweets.herokuapp.com [2] http://archive.org/download/wikitweets/wikitweets_archive.torrent [3] http://archive.org/download/wikitweets/wikitweets_archive.torrent [4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/
On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri taha.yaseri@gmail.com wrote:
My appreciation too. and the same question, do you also store the records?
bests, .t
On Thu, Apr 26, 2012 at 7:14 PM, emijrp emijrp@gmail.com wrote:
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
-- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam Personal website: https://sites.google.com/site/emijrp/
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Taha.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I love this tool so very much :) thank you!
On Wed, Sep 19, 2012 at 9:57 PM, Ed Summers ehs@pobox.com wrote:
Emilio, Taha:
I realize this was long enough ago that you may no longer be interested but I finally got around to adding an archive function to wikitweets [1]. Every time the app collects 1000 tweets that reference Wikipedia it dumps them to a file on Internet Archive [2].
One nice side effect of this is that you get a BitTorrent seed/peer for free [3], which makes mirroring the data pretty simple...if you have a BitTorrent client handy. I blogged a little bit about how it the archive function in wikitweets works [4].
Best, //Ed
[1] http://wikitweets.herokuapp.com [2] http://archive.org/download/wikitweets/wikitweets_archive.torrent [3] http://archive.org/download/wikitweets/wikitweets_archive.torrent [4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/
On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri taha.yaseri@gmail.com wrote:
My appreciation too. and the same question, do you also store the
records?
bests, .t
On Thu, Apr 26, 2012 at 7:14 PM, emijrp emijrp@gmail.com wrote:
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
-- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam Personal website: https://sites.google.com/site/emijrp/
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Taha.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Thank you Ed, That's something definitely worth-studying. Especially in order to have a less biased monitoring tool for "collective attention".
On Thu, Sep 20, 2012 at 10:23 AM, Samuel Klein meta.sj@gmail.com wrote:
I love this tool so very much :) thank you!
On Wed, Sep 19, 2012 at 9:57 PM, Ed Summers ehs@pobox.com wrote:
Emilio, Taha:
I realize this was long enough ago that you may no longer be interested but I finally got around to adding an archive function to wikitweets [1]. Every time the app collects 1000 tweets that reference Wikipedia it dumps them to a file on Internet Archive [2].
One nice side effect of this is that you get a BitTorrent seed/peer for free [3], which makes mirroring the data pretty simple...if you have a BitTorrent client handy. I blogged a little bit about how it the archive function in wikitweets works [4].
Best, //Ed
[1] http://wikitweets.herokuapp.com [2] http://archive.org/download/wikitweets/wikitweets_archive.torrent [3] http://archive.org/download/wikitweets/wikitweets_archive.torrent [4] http://inkdroid.org/journal/2012/09/19/archiving-wikitweets/
On Thu, Apr 26, 2012 at 6:28 PM, Taha Yasseri taha.yaseri@gmail.com wrote:
My appreciation too. and the same question, do you also store the
records?
bests, .t
On Thu, Apr 26, 2012 at 7:14 PM, emijrp emijrp@gmail.com wrote:
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
-- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam Personal website: https://sites.google.com/site/emijrp/
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Taha.
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
-- Samuel Klein @metasj w:user:sj +1 617 529 4266
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
At the moment there is no data store at play at all in the deployment on Heroku. Data is simply streamed from Twitter and Wikipedia and then delivered to any browsers who happens to be listening. It would be trivial to add a persistence layer, and some way of making the data available if there s a perceived need for the data. So it sounds like you might be interested?
On Thu, Apr 26, 2012 at 1:14 PM, emijrp emijrp@gmail.com wrote:
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
-- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam Personal website: https://sites.google.com/site/emijrp/
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
I would be very much interested, since such data shows us when and where people refer to WP in an overall image.
On Fri, Apr 27, 2012 at 3:16 AM, Ed Summers ehs@pobox.com wrote:
At the moment there is no data store at play at all in the deployment on Heroku. Data is simply streamed from Twitter and Wikipedia and then delivered to any browsers who happens to be listening. It would be trivial to add a persistence layer, and some way of making the data available if there s a perceived need for the data. So it sounds like you might be interested?
On Thu, Apr 26, 2012 at 1:14 PM, emijrp emijrp@gmail.com wrote:
2012/4/26 Ed Summers ehs@pobox.com
This is more on the experimental side of "research" but I just finished a prototype realtime visualization of tweets that reference Wikipedia:
Very cool. Do you archive the tweets or they are discarded?
-- Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com Pre-doctoral student at the University of Cádiz (Spain) Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam Personal website: https://sites.google.com/site/emijrp/
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Taha Yasseri, 27/04/2012 04:45:
I would be very much interested, since such data shows us when and where people refer to WP in an overall image.
Archiving tweets is surely useful, cf. http://archive.org/details/archiveteam-json-twitterstream-2012. An archive.org item with "wikitweets" in whatever format you grab them in (JSON? RSS?) should be quite trivial to do, and much appreciated.
Nemo
On Sat, Apr 28, 2012 at 5:08 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Archiving tweets is surely useful, cf. http://archive.org/details/archiveteam-json-twitterstream-2012. An archive.org item with "wikitweets" in whatever format you grab them in (JSON? RSS?) should be quite trivial to do, and much appreciated.
Good idea. I hadn't thought of using archive.org for this, and have been actually looking for an excuse to upload something to archive.org. Relying on the local filesystem on heroku is somewhat problematic [1], so I will probably need to run this archiving process elsewhere. Perhaps I'll get it running in wikimedia labs. Thanks for the suggestion!
//Ed
[1] https://devcenter.heroku.com/articles/java-faq#can_i_read_from_and_write_to_...
wiki-research-l@lists.wikimedia.org