WDQS URL shortener

List overview All Threads
Download

newer

older

another WDQS update lag example

SQID evolved

Dario Taraborelli

1 Jun 2016 1 Jun '16

7:23 a.m.

Hey all, while using shortened URLs from WDQS in the WikiCite report draft <https://meta.wikimedia.org/wiki/WikiCite_2016/Report>, it occurred to me that tinyurl.com is blacklisted on Meta. This is a major problem as it prevents concise URLs for gigantic queries from being linked from other Wikimedia wikis. Has anyone thought of this issue (Stas, Jonas?), in particular: should we ask Meta to remove the domain from the blacklist or potentially consider another URL shortening solution? Dario

Attachments:

attachment.htm (text/html — 639 bytes)

Show replies by date

Stas Malyshev

1 Jun 1 Jun

7:26 a.m.

Hi!

...

This is a major problem as it prevents concise URLs for gigantic queries from being linked from other Wikimedia wikis. Has anyone thought of this issue (Stas, Jonas?), in particular: should we ask Meta to remove the domain from the blacklist or potentially consider another URL shortening solution?

Yes, we thought of it (T112715) but since Wikimedia's own URL shortener (T108557) is not up yet, we have to use what is there. If you have an idea of a shortener more suitable than tinyurl.com, we could replace/add it. So far we didn't find a better alternative. -- Stas Malyshev smalyshev(a)wikimedia.org

Dario Taraborelli

8:33 a.m.

On Wed, Jun 1, 2016 at 9:26 AM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

...

Hi!

Yes, we thought of it (T112715) but since Wikimedia's own URL shortener (T108557) is not up yet, we have to use what is there.

got it

...

If you have an idea of a shortener more suitable than tinyurl.com, we could replace/add it. So far we didn't find a better alternative.

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

...

-- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- *Dario Taraborelli *Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter <http://twitter.com/readermeter>

Jonas Kress

8:59 a.m.

Hi! In theory phabricator [1] has an URL shortener, but it is not enabled. Cheers, Jons [1] https://phabricator.wikimedia.org/phurl/ 2016-06-01 10:33 GMT+02:00 Dario Taraborelli <dtaraborelli(a)wikimedia.org>rg>:

...

On Wed, Jun 1, 2016 at 9:26 AM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Hi!

Yes, we thought of it (T112715) but since Wikimedia's own URL shortener (T108557) is not up yet, we have to use what is there.

got it

If you have an idea of a shortener more suitable than tinyurl.com, we could replace/add it. So far we didn't find a better alternative.

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

-- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- *Dario Taraborelli *Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter <http://twitter.com/readermeter> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- Jonas KressSoftware Developer Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 BerlinPhone: +49 (0)30 219 158 26-0http://wikimedia.de Imagine a world, in which every single human being can freely share in the sum of all knowledge. That‘s our commitment. Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

Vi to

9:38 a.m.

Any time I find an url shorter I add it to global blacklist, same do other stewards/meta admins. That's because url shorters are the most common way to circumvent blacklisting. Vito 2016-06-01 10:33 GMT+02:00 Dario Taraborelli <dtaraborelli(a)wikimedia.org>rg>:

...

On Wed, Jun 1, 2016 at 9:26 AM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Hi!

Yes, we thought of it (T112715) but since Wikimedia's own URL shortener (T108557) is not up yet, we have to use what is there.

got it

If you have an idea of a shortener more suitable than tinyurl.com, we could replace/add it. So far we didn't find a better alternative.

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

-- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Andrew Gray

9:55 a.m.

Hi Dario (et al), I'm not deeply familiar with the spam blacklist issues (haven't worked with it for some years) but the problem I can see is that if we pick a currently OK third-party shortener and allow that, it'll be incredibly tempting for people to start using it for spam... If we're wanting to have a shortener, it'd probably need to have fairly restricted inputs to avoid this problem, which comes straight back round to "create our own and restrict what people can do with it". Not sure there's an easy solution to this. Andrew. On 1 June 2016 at 09:33, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

On Wed, Jun 1, 2016 at 9:26 AM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

Hi!

Yes, we thought of it (T112715) but since Wikimedia's own URL shortener (T108557) is not up yet, we have to use what is there.

got it

If you have an idea of a shortener more suitable than tinyurl.com, we could replace/add it. So far we didn't find a better alternative.

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

-- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- Dario Taraborelli Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- - Andrew Gray andrew.gray(a)dunelm.org.uk

Federico Leva (Nemo)

noon

Dario Taraborelli, 01/06/2016 10:33:

...

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

Nearly all URL shorteners get blacklisted, eventually. Nemo

Dario Taraborelli

12:17 p.m.

On Wed, Jun 1, 2016 at 2:00 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

...

Dario Taraborelli, 01/06/2016 10:33:

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

Nearly all URL shorteners get blacklisted, eventually.

that makes sense. It sounds like in the short term (and until we have a Wikimedia-operated shortener), using full URLs from WDQS is – alas – the only way to go. One option we haven't mentioned would be for WDQS itself to support URL shortening, I have no idea where that would sit in terms of priorities. Dario

...

Nemo _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- *Dario Taraborelli *Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter <http://twitter.com/readermeter>

Julie McMurry

4:39 p.m.

I suggest considering w3id Julie On Wednesday, June 1, 2016, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

On Wed, Jun 1, 2016 at 2:00 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com <javascript:_e(%7B%7D,'cvml','nemowiki@gmail.com');>> wrote:

Dario Taraborelli, 01/06/2016 10:33:

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

Nearly all URL shorteners get blacklisted, eventually.

Nemo _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <javascript:_e(%7B%7D,'cvml','Wikidata@lists.wikimedia.org');> https://lists.wikimedia.org/mailman/listinfo/wikidata

-- *Dario Taraborelli *Head of Research, Wikimedia Foundation wikimediafoundation.org • nitens.org • @readermeter <http://twitter.com/readermeter>

-- ---------------------------------- mcmurry.julie(a)gmail.com +1 805 455 5877 skype: mcmurry.julie git: jmcmurry ----------------------------------

Brill Lyle

4:58 p.m.

For many events at Wikimedia NYC, we use a shortcut -- once you're on Wikipedia -- like "WP:Harlem" also known as "Wikipedia:Harlem" The redirect (before it redirects): https://en.wikipedia.org/w/index.php?title=Wikipedia:Harlem&redirect=no Redirects to the Schomburg editathon we held earlier this year https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Schomburg I understand why URL shorteners are non ideal. I support that I guess. These Wikipedia shortcuts have been very helpful although they probably don't work across projects. Another reason to avoid Meta #sigh Best, - Erika *Erika Herzog* Wikipedia *User:BrillLyle* <https://en.wikipedia.org/wiki/User:BrillLyle> Secretary, Wikimedia NYC <https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC> On Wed, Jun 1, 2016 at 8:17 AM, Dario Taraborelli < dtaraborelli(a)wikimedia.org> wrote:

...

On Wed, Jun 1, 2016 at 2:00 PM, Federico Leva (Nemo) <nemowiki(a)gmail.com> wrote:

Dario Taraborelli, 01/06/2016 10:33:

I don't, it probably depends on what shorteners are most used for spam purposes across Wikimedia projects. Maybe someone familiar with URL blacklisting from major wikis can comment?

Nearly all URL shorteners get blacklisted, eventually.

Nemo _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Stas Malyshev

10:23 p.m.

Hi!

...

Julie McMurry

2 Jun 2 Jun

1:11 a.m.

Hi there, may I ask what link shorteners provide you that w3id does not? Eg baked in metrics or 10char urls? Just curious why you would want to reimplement. Julie On Wednesday, June 1, 2016, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

...

Hi!

I don't think it'd be very high. It would require building infrastructure that Wiki extension already has, and for sole purpose of implementing functionality that is already provided by that extension. So I think the best way here is just to wait until it's deployed, and maybe gently prod responsible people from time to time :) -- Stas Malyshev smalyshev(a)wikimedia.org <javascript:;> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <javascript:;> https://lists.wikimedia.org/mailman/listinfo/wikidata

-- ---------------------------------- mcmurry.julie(a)gmail.com +1 805 455 5877 skype: mcmurry.julie git: jmcmurry ----------------------------------

Stas Malyshev

3:58 a.m.

Hi!

...

Hi there, may I ask what link shorteners provide you that w3id does not? Eg baked in metrics or 10char urls? Just curious why you would want to reimplement.

...

From what I can see, the target audience of w3id is a relatively small

set of very stable URL prefixes that are used a lot and never change. It also does not aim at making these URLs shorter, it aims at making them stable. Adding URL namespace to it is a manual process, and individual URLs are not stored. The target of URL shorteners is much bigger set of URLs, many of which are relatively low-use or transient, but which can be created via automatic means in great volumes, which make URL shorter and which are aimed at storing, at least for a while, each URL as individual data piece. So, for our purposes w3id would not be very useful. -- Stas Malyshev smalyshev(a)wikimedia.org

Julie McMurry

4:22 a.m.

While I agree the primary aim isn't shortening, the result is usually much shorter by virtue of cutting out everything non essential to identification. I say this as a non stakeholder in w3id, just a fan. Nevertheless I take your point that it isn't automated. I would like to understand the use case better; could you describe a specific scenario? Thanks Julie On Wednesday, June 1, 2016, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

...

Hi!

Hi there, may I ask what link shorteners provide you that w3id does not? Eg baked in metrics or 10char urls? Just curious why you would want to reimplement.

From what I can see, the target audience of w3id is a relatively small set of very stable URL prefixes that are used a lot and never change. It also does not aim at making these URLs shorter, it aims at making them stable. Adding URL namespace to it is a manual process, and individual URLs are not stored. The target of URL shorteners is much bigger set of URLs, many of which are relatively low-use or transient, but which can be created via automatic means in great volumes, which make URL shorter and which are aimed at storing, at least for a while, each URL as individual data piece. So, for our purposes w3id would not be very useful. -- Stas Malyshev smalyshev(a)wikimedia.org <javascript:;> _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org <javascript:;> https://lists.wikimedia.org/mailman/listinfo/wikidata

-- ---------------------------------- mcmurry.julie(a)gmail.com +1 805 455 5877 skype: mcmurry.julie git: jmcmurry ----------------------------------

Tom Morris

4:51 a.m.

On Thu, Jun 2, 2016 at 12:22 AM, Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

...

While I agree the primary aim isn't shortening, the result is usually much shorter by virtue of cutting out everything non essential to identification.

Except in the case of a giant query string, such as a complex SPARQL query.

Benjamin Good

3 Jun 3 Jun

4:01 p.m.

I don't want to suggest a technology, but the URL shortener on wdqs has been EXTREMELY USEFUL to me and it would be major bummer to lose it. I recently taught a class and used a variety of examples of SPARQL queries over wikidata. Having that shortener made it much faster for me to assemble the lecture and give people easy ways to get to those queries. I could have used another one on my own I guess, but the current implementation is much faster and less error prone when dealing with monster sparqling urls... please find a way to keep it -Ben On Wed, Jun 1, 2016 at 9:51 PM, Tom Morris <tfmorris(a)gmail.com> wrote:

...

On Thu, Jun 2, 2016 at 12:22 AM, Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

While I agree the primary aim isn't shortening, the result is usually much shorter by virtue of cutting out everything non essential to identification.

Except in the case of a giant query string, such as a complex SPARQL query. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Stas Malyshev

4:26 p.m.

Hi!

...

I could have used another one on my own I guess, but the current implementation is much faster and less error prone when dealing with monster sparqling urls... please find a way to keep it

Benjamin Good

5:30 p.m.

awesome, thank you! In case its useful for anyone, I was using wikidata to teach biologists and chemists about knowledge graphs (many tinyurls toward the end) http://www.slideshare.net/goodb/computing-on-the-shoulders-of-giants On the second part of the course students were provided with the following Jupyter python notebook which runs a sparql query against wikidata and generates an output file suitable for loading in Cytoscape (a commonly used network visualization tool in biology). Could be useful for others that need to teach this stuff.. https://github.com/SuLab/sparql_to_pandas/blob/master/SPARQL_pandas.ipynb On Fri, Jun 3, 2016 at 9:26 AM, Stas Malyshev <smalyshev(a)wikimedia.org> wrote:

...

Hi!

I could have used another one on my own I guess, but the current implementation is much faster and less error prone when dealing with monster sparqling urls... please find a way to keep it

There are no plans to remove URL shortening. There are plans to switch URL shortener to Wikimedia's own one, which is supposed to be coming up eventually, but before that, we plan to use existing ones. We might change a provider if it turns out there is a better one, but we do not plan to remove the functionality. -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Julie McMurry

4:17 p.m.

Makes perfect sense now, thanks for explaining. Julie On Thu, Jun 2, 2016 at 12:51 AM, Tom Morris <tfmorris(a)gmail.com> wrote:

...

On Thu, Jun 2, 2016 at 12:22 AM, Julie McMurry <mcmurry.julie(a)gmail.com> wrote:

While I agree the primary aim isn't shortening, the result is usually much shorter by virtue of cutting out everything non essential to identification.

-- ---------------------------------- mcmurry.julie(a)gmail.com +1 805 455 5877 skype: mcmurry.julie git: jmcmurry ----------------------------------

2898

days inactive

2900

days old

wikidata@lists.wikimedia.org

Manage subscription

18 comments

10 participants

tags (0)

participants (10)

Andrew Gray
Benjamin Good
Brill Lyle
Dario Taraborelli
Federico Leva (Nemo)
Jonas Kress
Julie McMurry
Stas Malyshev
Tom Morris
Vi to