[Labs-l] Labs-l Digest, Vol 39, Issue 13

John phoenixoverride at gmail.com
Fri Mar 13 19:08:12 UTC 2015 

OK, No offense here, your not being helpful. Are you pulling from a
category, using page text or some format already in the database? The
reason I am asking is that depending on how you are selecting the articles,
it might be possible to match query the database in a manor that optimizes
the process and makes the overall time drop drastically. There have been a
few quires that I have played with in the past that originally took hours
or even days, and we where able to get them down to  a few minutes. However
without more information the vague data we are given so far isn't that
helpful.

On Fri, Mar 13, 2015 at 3:01 PM, Marc Miquel <marcmiquel at gmail.com> wrote:

> I get them according to some selection I do according to other parameters
> more related to the content. The selection of these 300000, which could be
> either 30000 or even 500000 for other cases (like german wiki) is not an
> issue. The link analysis to see if these 300.000 receive links from another
> group of articles is my concern...
>
> Marc
> ᐧ
>
> 2015-03-13 19:56 GMT+01:00 John <phoenixoverride at gmail.com>:
>
>> where are you getting the list of 300k pages from? I want to get a feel
>> for the kinds of queries your running so that we can optimize the process
>> for you.
>>
>> On Fri, Mar 13, 2015 at 2:53 PM, Marc Miquel <marcmiquel at gmail.com>
>> wrote:
>>
>>> I load from a file "page_titles" and "page_ids" and put them in a
>>> dictionary. One option I haven't used would be putting than into a database
>>> and INNER Joining with the pagelinks table to just obtain the links for
>>> those articles. Still, if the list is 300.000, even this is just 20% of the
>>> database, it is still a lot.
>>>
>>> Marc
>>> ᐧ
>>>
>>> 2015-03-13 19:51 GMT+01:00 John <phoenixoverride at gmail.com>:
>>>
>>>> Where are you getting your list of pages from?
>>>>
>>>> On Fri, Mar 13, 2015 at 2:46 PM, Marc Miquel <marcmiquel at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi John,
>>>>>
>>>>>
>>>>> My queries are to obtain "inlinks" and "outlinks" for some articles I
>>>>> have in a group (x). Then I check (using python) if they have inlinks and
>>>>> outlinks from another group of articles. By now I am doing a query for each
>>>>> article. I wanted to obtain all links for group (x) and then do this
>>>>> comprovation....But getting all links for groups as big as 300000 articles
>>>>> would imply 6 million links. Is it possible to obtain all this or is there
>>>>> a MySQL/RAM limit?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Marc
>>>>>
>>>>> ᐧ
>>>>>
>>>>> 2015-03-13 19:29 GMT+01:00 <labs-l-request at lists.wikimedia.org>:
>>>>>
>>>>>> Send Labs-l mailing list submissions to
>>>>>>         labs-l at lists.wikimedia.org
>>>>>>
>>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>>>         https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>>> or, via email, send a message with subject or body 'help' to
>>>>>>         labs-l-request at lists.wikimedia.org
>>>>>>
>>>>>> You can reach the person managing the list at
>>>>>>         labs-l-owner at lists.wikimedia.org
>>>>>>
>>>>>> When replying, please edit your Subject line so it is more specific
>>>>>> than "Re: Contents of Labs-l digest..."
>>>>>>
>>>>>>
>>>>>> Today's Topics:
>>>>>>
>>>>>>    1. dimension well my queries for very large tables like
>>>>>>       pagelinks - Tool Labs (Marc Miquel)
>>>>>>    2. Re: dimension well my queries for very large tables like
>>>>>>       pagelinks - Tool Labs (John)
>>>>>>    3. Re: Questions regarding the Labs Terms of use (Tim Landscheidt)
>>>>>>    4. Re: Questions regarding the Labs Terms of use (Ryan Lane)
>>>>>>    5. Re: Questions regarding the Labs Terms of use (Pine W)
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>>
>>>>>> Message: 1
>>>>>> Date: Fri, 13 Mar 2015 17:59:09 +0100
>>>>>> From: Marc Miquel <marcmiquel at gmail.com>
>>>>>> To: "labs-l at lists.wikimedia.org" <labs-l at lists.wikimedia.org>
>>>>>> Subject: [Labs-l] dimension well my queries for very large tables like
>>>>>>         pagelinks - Tool Labs
>>>>>> Message-ID:
>>>>>>         <CANSEGinZBWYsb0Y9r9Yk8AZo3COwzT4NTs7YkFxj=
>>>>>> naEa9d6+w at mail.gmail.com>
>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>
>>>>>> Hello guys,
>>>>>>
>>>>>> I have a question regarding Tool Labs. I am doing research on links
>>>>>> and
>>>>>> although I know very well what I am looking for I struggle in how to
>>>>>> get it
>>>>>> effectively...
>>>>>>
>>>>>> I need to know your opinion because you know very well the system and
>>>>>> what's feasible and what is not.
>>>>>>
>>>>>> I explain you what I need to do:
>>>>>> I have a list of articles for different languages which I need to
>>>>>> check
>>>>>> their pagelinks and see where they point to and from where they point
>>>>>> at
>>>>>> them.
>>>>>>
>>>>>> I now do a query for each article id in this list of articles, which
>>>>>> goes
>>>>>> from 80000 in some wikipedias to 300000 in other and more. I have to
>>>>>> do it
>>>>>> several times and it is very time consuming (several days). I wish I
>>>>>> could
>>>>>> only count the total of links for each case but I need to see only
>>>>>> some of
>>>>>> the links per article.
>>>>>>
>>>>>> I was thinking about getting all pagelinks and iterating using python
>>>>>> (which is the language I use for all this). This would be much faster
>>>>>> because I'd save all the queries, one per article, I am doing now. But
>>>>>> pagelinks table has millions of rows and I cannot load that because
>>>>>> mysql
>>>>>> would die. I could buffer, but I haven't tried if it works also.
>>>>>>
>>>>>> I am considering creating a personal table in the database with
>>>>>> titles,
>>>>>> ids, and inner joining to just obtain the pagelinks for these 300.000
>>>>>> articles. With this I would just retrieve 20% of the database instead
>>>>>> of
>>>>>> the 100%. That would be maybe 8M rows sometimes (page_title or
>>>>>> page_id, one
>>>>>> of both per row), or even more... loaded into python dictionaries and
>>>>>> lists. Would that be a problem...? I have no idea of how much RAM this
>>>>>> implies and how much I can use in Tool labs.
>>>>>>
>>>>>> I am totally lost when I get these problems related to scale...I
>>>>>> thought
>>>>>> about writing to the IRC channel but I thought it was maybe too long
>>>>>> and
>>>>>> too specific. If you give me any hint that would really help.
>>>>>>
>>>>>> Thank you very much!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Marc Miquel
>>>>>> ᐧ
>>>>>> -------------- next part --------------
>>>>>> An HTML attachment was scrubbed...
>>>>>> URL: <
>>>>>> https://lists.wikimedia.org/pipermail/labs-l/attachments/20150313/59a09113/attachment-0001.html
>>>>>> >
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> Message: 2
>>>>>> Date: Fri, 13 Mar 2015 13:07:20 -0400
>>>>>> From: John <phoenixoverride at gmail.com>
>>>>>> To: Wikimedia Labs <labs-l at lists.wikimedia.org>
>>>>>> Subject: Re: [Labs-l] dimension well my queries for very large tables
>>>>>>         like pagelinks - Tool Labs
>>>>>> Message-ID:
>>>>>>         <CAP-JHpn=ToVYdT7i-imp7+XLTkgQ1PtieORx7BTuVAJw=
>>>>>> YSbFQ at mail.gmail.com>
>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>
>>>>>> what kind of queries are you doing? odds are they can be optimized.
>>>>>>
>>>>>> On Fri, Mar 13, 2015 at 12:59 PM, Marc Miquel <marcmiquel at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hello guys,
>>>>>> >
>>>>>> > I have a question regarding Tool Labs. I am doing research on links
>>>>>> and
>>>>>> > although I know very well what I am looking for I struggle in how
>>>>>> to get it
>>>>>> > effectively...
>>>>>> >
>>>>>> > I need to know your opinion because you know very well the system
>>>>>> and
>>>>>> > what's feasible and what is not.
>>>>>> >
>>>>>> > I explain you what I need to do:
>>>>>> > I have a list of articles for different languages which I need to
>>>>>> check
>>>>>> > their pagelinks and see where they point to and from where they
>>>>>> point at
>>>>>> > them.
>>>>>> >
>>>>>> > I now do a query for each article id in this list of articles,
>>>>>> which goes
>>>>>> > from 80000 in some wikipedias to 300000 in other and more. I have
>>>>>> to do it
>>>>>> > several times and it is very time consuming (several days). I wish
>>>>>> I could
>>>>>> > only count the total of links for each case but I need to see only
>>>>>> some of
>>>>>> > the links per article.
>>>>>> >
>>>>>> > I was thinking about getting all pagelinks and iterating using
>>>>>> python
>>>>>> > (which is the language I use for all this). This would be much
>>>>>> faster
>>>>>> > because I'd save all the queries, one per article, I am doing now.
>>>>>> But
>>>>>> > pagelinks table has millions of rows and I cannot load that because
>>>>>> mysql
>>>>>> > would die. I could buffer, but I haven't tried if it works also.
>>>>>> >
>>>>>> > I am considering creating a personal table in the database with
>>>>>> titles,
>>>>>> > ids, and inner joining to just obtain the pagelinks for these
>>>>>> 300.000
>>>>>> > articles. With this I would just retrieve 20% of the database
>>>>>> instead of
>>>>>> > the 100%. That would be maybe 8M rows sometimes (page_title or
>>>>>> page_id, one
>>>>>> > of both per row), or even more... loaded into python dictionaries
>>>>>> and
>>>>>> > lists. Would that be a problem...? I have no idea of how much RAM
>>>>>> this
>>>>>> > implies and how much I can use in Tool labs.
>>>>>> >
>>>>>> > I am totally lost when I get these problems related to scale...I
>>>>>> thought
>>>>>> > about writing to the IRC channel but I thought it was maybe too
>>>>>> long and
>>>>>> > too specific. If you give me any hint that would really help.
>>>>>> >
>>>>>> > Thank you very much!
>>>>>> >
>>>>>> > Cheers,
>>>>>> >
>>>>>> > Marc Miquel
>>>>>> > ᐧ
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Labs-l mailing list
>>>>>> > Labs-l at lists.wikimedia.org
>>>>>> > https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>>> >
>>>>>> >
>>>>>> -------------- next part --------------
>>>>>> An HTML attachment was scrubbed...
>>>>>> URL: <
>>>>>> https://lists.wikimedia.org/pipermail/labs-l/attachments/20150313/c60b6d44/attachment-0001.html
>>>>>> >
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> Message: 3
>>>>>> Date: Fri, 13 Mar 2015 17:36:00 +0000
>>>>>> From: Tim Landscheidt <tim at tim-landscheidt.de>
>>>>>> To: labs-l at lists.wikimedia.org
>>>>>> Subject: Re: [Labs-l] Questions regarding the Labs Terms of use
>>>>>> Message-ID: <878uf0vlz3.fsf at passepartout.tim-landscheidt.de>
>>>>>> Content-Type: text/plain
>>>>>>
>>>>>> (anonymous) wrote:
>>>>>>
>>>>>> > [...]
>>>>>>
>>>>>> > To be clear: I'm not going to make my code proprietary in
>>>>>> > any way. I just wanted to know whether I'm entitled to ask
>>>>>> > for the source of every Labs bot ;-)
>>>>>>
>>>>>> Everyone is entitled to /ask/, but I don't think you have a
>>>>>> right to /receive/ the source :-).
>>>>>>
>>>>>> AFAIK, there are two main reasons for the clause:
>>>>>>
>>>>>> a) WMF doesn't want to have to deal with individual licences
>>>>>>    that may or may not have the potential for litigation
>>>>>>    ("The Software shall be used for Good, not Evil").  With
>>>>>>    requiring OSI-approved, tried and true licences, the risk
>>>>>>    is negligible.
>>>>>>
>>>>>> b) Bots and tools running on an infrastructure financed by
>>>>>>    donors, like contributions to Wikipedia & Co., shouldn't
>>>>>>    be usable for blackmail.  Noone should be in a legal po-
>>>>>>    sition to demand something "or else ..."  The perpetuity
>>>>>>    of OS licences guarantees that everyone can be truly
>>>>>>    thankful to developers without having to fear that other-
>>>>>>    wise they shut down devices, delete content, etc.
>>>>>>
>>>>>> But the nice thing about collaboratively developed open
>>>>>> source software is that it usually is of a better quality,
>>>>>> so clandestine code is often not that interesting.
>>>>>>
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> Message: 4
>>>>>> Date: Fri, 13 Mar 2015 11:52:18 -0600
>>>>>> From: Ryan Lane <rlane32 at gmail.com>
>>>>>> To: Wikimedia Labs <labs-l at lists.wikimedia.org>
>>>>>> Subject: Re: [Labs-l] Questions regarding the Labs Terms of use
>>>>>> Message-ID:
>>>>>>         <
>>>>>> CALKgCA3Lv-SQoeibEsm7Ckc0gaPJwph_b0HSTx+actaKMDuXmg at mail.gmail.com>
>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>
>>>>>> On Fri, Mar 13, 2015 at 8:42 AM, Ricordisamoa <
>>>>>> ricordisamoa at openmailbox.org>
>>>>>> wrote:
>>>>>>
>>>>>> > From https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use
>>>>>> > (verbatim): "Do not use or install any software unless the software
>>>>>> is
>>>>>> > licensed under an Open Source license".
>>>>>> > What about tools and services made up of software themselves? Do
>>>>>> they have
>>>>>> > to be Open Source?
>>>>>> > Strictly speaking, do the Terms of use require that all code be made
>>>>>> > available to the public?
>>>>>> > Thanks in advance.
>>>>>> >
>>>>>> >
>>>>>> As the person who wrote the initial terms and included this I can
>>>>>> speak to
>>>>>> the spirit of the term (I'm not a lawyer, so I won't try to go into
>>>>>> any
>>>>>> legal issues).
>>>>>>
>>>>>> I created Labs with the intent that it could be used as a mechanism
>>>>>> to fork
>>>>>> the projects as a whole, if necessary. A means to this end was
>>>>>> including
>>>>>> non-WMF employees in the process of infrastructure operations (which
>>>>>> is
>>>>>> outside the goals of the tools project in Labs). Tools/services that
>>>>>> are
>>>>>> can't be distributed publicly harm that goal. Tools/services that
>>>>>> aren't
>>>>>> open source completely break that goal. It's fine if you wish to not
>>>>>> maintain the code in a public git repo, but if another tool maintainer
>>>>>> wishes to publish your code, there should be nothing blocking that.
>>>>>>
>>>>>> Depending on external closed source services is a debatable topic. I
>>>>>> know
>>>>>> in the past we've decided to allow it. It goes against the spirit of
>>>>>> the
>>>>>> project, but it doesn't require us to distribute close sourced
>>>>>> software in
>>>>>> the case of a fork.
>>>>>>
>>>>>> My personal opinion is that your code should be in a public
>>>>>> repository to
>>>>>> encourage collaboration. As the terms are written, though, your code
>>>>>> is
>>>>>> required to be open source, and any libraries it depends on must be
>>>>>> as well.
>>>>>>
>>>>>> - Ryan
>>>>>> -------------- next part --------------
>>>>>> An HTML attachment was scrubbed...
>>>>>> URL: <
>>>>>> https://lists.wikimedia.org/pipermail/labs-l/attachments/20150313/8d760f03/attachment-0001.html
>>>>>> >
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> Message: 5
>>>>>> Date: Fri, 13 Mar 2015 11:29:47 -0700
>>>>>> From: Pine W <wiki.pine at gmail.com>
>>>>>> To: Wikimedia Labs <labs-l at lists.wikimedia.org>
>>>>>> Subject: Re: [Labs-l] Questions regarding the Labs Terms of use
>>>>>> Message-ID:
>>>>>>         <CAF=dyJjO69O-ye+327BU6wC_k_+AQvwUq0rfZvW0YaV=
>>>>>> P+iCaA at mail.gmail.com>
>>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>>
>>>>>> Question: are there heightened security or privacy risks posed by
>>>>>> having
>>>>>> non-open-source code running in Labs?
>>>>>>
>>>>>> Is anyone proactively auditing Labs software for open source
>>>>>> compliance,
>>>>>> and if not, should this be done?
>>>>>>
>>>>>> Pine
>>>>>> On Mar 13, 2015 10:52 AM, "Ryan Lane" <rlane32 at gmail.com> wrote:
>>>>>>
>>>>>> > On Fri, Mar 13, 2015 at 8:42 AM, Ricordisamoa <
>>>>>> > ricordisamoa at openmailbox.org> wrote:
>>>>>> >
>>>>>> >> From
>>>>>> https://wikitech.wikimedia.org/wiki/Wikitech:Labs_Terms_of_use
>>>>>> >> (verbatim): "Do not use or install any software unless the
>>>>>> software is
>>>>>> >> licensed under an Open Source license".
>>>>>> >> What about tools and services made up of software themselves? Do
>>>>>> they
>>>>>> >> have to be Open Source?
>>>>>> >> Strictly speaking, do the Terms of use require that all code be
>>>>>> made
>>>>>> >> available to the public?
>>>>>> >> Thanks in advance.
>>>>>> >>
>>>>>> >>
>>>>>> > As the person who wrote the initial terms and included this I can
>>>>>> speak to
>>>>>> > the spirit of the term (I'm not a lawyer, so I won't try to go into
>>>>>> any
>>>>>> > legal issues).
>>>>>> >
>>>>>> > I created Labs with the intent that it could be used as a mechanism
>>>>>> to
>>>>>> > fork the projects as a whole, if necessary. A means to this end was
>>>>>> > including non-WMF employees in the process of infrastructure
>>>>>> operations
>>>>>> > (which is outside the goals of the tools project in Labs).
>>>>>> Tools/services
>>>>>> > that are can't be distributed publicly harm that goal.
>>>>>> Tools/services that
>>>>>> > aren't open source completely break that goal. It's fine if you
>>>>>> wish to not
>>>>>> > maintain the code in a public git repo, but if another tool
>>>>>> maintainer
>>>>>> > wishes to publish your code, there should be nothing blocking that.
>>>>>> >
>>>>>> > Depending on external closed source services is a debatable topic.
>>>>>> I know
>>>>>> > in the past we've decided to allow it. It goes against the spirit
>>>>>> of the
>>>>>> > project, but it doesn't require us to distribute close sourced
>>>>>> software in
>>>>>> > the case of a fork.
>>>>>> >
>>>>>> > My personal opinion is that your code should be in a public
>>>>>> repository to
>>>>>> > encourage collaboration. As the terms are written, though, your
>>>>>> code is
>>>>>> > required to be open source, and any libraries it depends on must be
>>>>>> as well.
>>>>>> >
>>>>>> > - Ryan
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Labs-l mailing list
>>>>>> > Labs-l at lists.wikimedia.org
>>>>>> > https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>>> >
>>>>>> >
>>>>>> -------------- next part --------------
>>>>>> An HTML attachment was scrubbed...
>>>>>> URL: <
>>>>>> https://lists.wikimedia.org/pipermail/labs-l/attachments/20150313/fc853f84/attachment.html
>>>>>> >
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> Labs-l mailing list
>>>>>> Labs-l at lists.wikimedia.org
>>>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>>>
>>>>>>
>>>>>> End of Labs-l Digest, Vol 39, Issue 13
>>>>>> **************************************
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150313/0c34357d/attachment-0001.html>

More information about the Labs-l mailing list