Hello everyone,
I wanted to ask about resource allocation on stat1005. We https://meta.wikimedia.org/wiki/Research:Understanding_Wikidata_Queries need quite a bit since we process every entry in wdqs_extract and I was wondering how many cores and how much memory we can use without conflicting with anyone else.
Greetings,
Adrian
I will soon start the two Wikistats jobs which run for about several weeks each month,
They might use two cores each, one for unzip, one for perl.
How many cores are there anyway?
Cheers,
Erik
From: Analytics [mailto:analytics-bounces@lists.wikimedia.org] On Behalf Of Adrian Bielefeldt Sent: Saturday, August 12, 2017 19:44 To: analytics@lists.wikimedia.org Subject: [Analytics] Resources stat1005
Hello everyone,
I wanted to ask about resource allocation on stat1005. We https://meta.wikimedia.org/wiki/Research:Understanding_Wikidata_Queries need quite a bit since we process every entry in wdqs_extract and I was wondering how many cores and how much memory we can use without conflicting with anyone else.
Greetings,
Adrian
I have only 2 comments:
1. Please nice https://en.wikipedia.org/wiki/Nice_(Unix) any heavy long running local processes, so that others can continue to use the machine.
2. For large data, consider using the Hadoop cluster! I think you are getting your data from the webrequest logs in Hadoop anyway, so you might as well continue to do processing there, no? If you do, you shouldn’t have to worry (too much) about resource contention: https://yarn.wikimedia.org/cluster/scheduler
:)
- Andrew Otto Systems Engineer, WMF
On Sat, Aug 12, 2017 at 2:20 PM, Erik Zachte ezachte@wikimedia.org wrote:
I will soon start the two Wikistats jobs which run for about several weeks each month,
They might use two cores each, one for unzip, one for perl.
How many cores are there anyway?
Cheers,
Erik
*From:* Analytics [mailto:analytics-bounces@lists.wikimedia.org] *On Behalf Of *Adrian Bielefeldt *Sent:* Saturday, August 12, 2017 19:44 *To:* analytics@lists.wikimedia.org *Subject:* [Analytics] Resources stat1005
Hello everyone,
I wanted to ask about resource allocation on stat1005. We https://meta.wikimedia.org/wiki/Research:Understanding_Wikidata_Queries need quite a bit since we process every entry in wdqs_extract and I was wondering how many cores and how much memory we can use without conflicting with anyone else.
Greetings,
Adrian
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi Andrew,
thanks for the advice. Quick follow-up question: Which kind of access/account do I need for yarn.wikimedia.org? Neither my MediaWiki-Account nor my WikiTech-Account work.
Greetings,
Adrian
On 08/12/2017 09:58 PM, Andrew Otto wrote:
I have only 2 comments:
- Please nice https://en.wikipedia.org/wiki/Nice_%28Unix%29 any
heavy long running local processes, so that others can continue to use the machine.
- For large data, consider using the Hadoop cluster! I think you are
getting your data from the webrequest logs in Hadoop anyway, so you might as well continue to do processing there, no? If you do, you shouldn’t have to worry (too much) about resource contention: https://yarn.wikimedia.org/cluster/scheduler
:)
- Andrew Otto Systems Engineer, WMF
On Sat, Aug 12, 2017 at 2:20 PM, Erik Zachte <ezachte@wikimedia.org mailto:ezachte@wikimedia.org> wrote:
I will soon start the two Wikistats jobs which run for about several weeks each month, They might use two cores each, one for unzip, one for perl. How many cores are there anyway? Cheers, Erik *From:*Analytics [mailto:analytics-bounces@lists.wikimedia.org <mailto:analytics-bounces@lists.wikimedia.org>] *On Behalf Of *Adrian Bielefeldt *Sent:* Saturday, August 12, 2017 19:44 *To:* analytics@lists.wikimedia.org <mailto:analytics@lists.wikimedia.org> *Subject:* [Analytics] Resources stat1005 Hello everyone, I wanted to ask about resource allocation on stat1005. We <https://meta.wikimedia.org/wiki/Research:Understanding_Wikidata_Queries> need quite a bit since we process every entry in wdqs_extract and I was wondering how many cores and how much memory we can use without conflicting with anyone else. Greetings, Adrian _______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org <mailto:Analytics@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/analytics <https://lists.wikimedia.org/mailman/listinfo/analytics>
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Adrian,
You already have access to use the cluster, which is where you should move your processing, the link to yarn was just to show resource consumption.
Thanks,
Nuria
On Sat, Aug 12, 2017 at 3:52 PM, Adrian Bielefeldt < Adrian.Bielefeldt@mailbox.tu-dresden.de> wrote:
Hi Andrew,
thanks for the advice. Quick follow-up question: Which kind of access/account do I need for yarn.wikimedia.org? Neither my MediaWiki-Account nor my WikiTech-Account work.
Greetings,
Adrian
On 08/12/2017 09:58 PM, Andrew Otto wrote:
I have only 2 comments:
- Please nice https://en.wikipedia.org/wiki/Nice_%28Unix%29 any heavy
long running local processes, so that others can continue to use the machine.
- For large data, consider using the Hadoop cluster! I think you are
getting your data from the webrequest logs in Hadoop anyway, so you might as well continue to do processing there, no? If you do, you shouldn’t have to worry (too much) about resource contention: https://yarn. wikimedia.org/cluster/scheduler
:)
- Andrew Otto Systems Engineer, WMF
On Sat, Aug 12, 2017 at 2:20 PM, Erik Zachte ezachte@wikimedia.org wrote:
I will soon start the two Wikistats jobs which run for about several weeks each month,
They might use two cores each, one for unzip, one for perl.
How many cores are there anyway?
Cheers,
Erik
*From:* Analytics [mailto:analytics-bounces@lists.wikimedia.org] *On Behalf Of *Adrian Bielefeldt *Sent:* Saturday, August 12, 2017 19:44 *To:* analytics@lists.wikimedia.org *Subject:* [Analytics] Resources stat1005
Hello everyone,
I wanted to ask about resource allocation on stat1005. We https://meta.wikimedia.org/wiki/Research:Understanding_Wikidata_Queries need quite a bit since we process every entry in wdqs_extract and I was wondering how many cores and how much memory we can use without conflicting with anyone else.
Greetings,
Adrian
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing listAnalytics@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi Adrian,
you should open a phab task like the following: https://phabricator.wikimedia.org/T158053 to get into the nda LDAP group (if you really need it as Nuria mentioned :).
Luca
2017-08-13 0:52 GMT+02:00 Adrian Bielefeldt < Adrian.Bielefeldt@mailbox.tu-dresden.de>:
Hi Andrew,
thanks for the advice. Quick follow-up question: Which kind of access/account do I need for yarn.wikimedia.org? Neither my MediaWiki-Account nor my WikiTech-Account work.
Greetings,
Adrian
On 08/12/2017 09:58 PM, Andrew Otto wrote:
I have only 2 comments:
- Please nice https://en.wikipedia.org/wiki/Nice_%28Unix%29 any heavy
long running local processes, so that others can continue to use the machine.
- For large data, consider using the Hadoop cluster! I think you are
getting your data from the webrequest logs in Hadoop anyway, so you might as well continue to do processing there, no? If you do, you shouldn’t have to worry (too much) about resource contention: https://yarn. wikimedia.org/cluster/scheduler
:)
- Andrew Otto Systems Engineer, WMF
On Sat, Aug 12, 2017 at 2:20 PM, Erik Zachte ezachte@wikimedia.org wrote:
I will soon start the two Wikistats jobs which run for about several weeks each month,
They might use two cores each, one for unzip, one for perl.
How many cores are there anyway?
Cheers,
Erik
*From:* Analytics [mailto:analytics-bounces@lists.wikimedia.org] *On Behalf Of *Adrian Bielefeldt *Sent:* Saturday, August 12, 2017 19:44 *To:* analytics@lists.wikimedia.org *Subject:* [Analytics] Resources stat1005
Hello everyone,
I wanted to ask about resource allocation on stat1005. We https://meta.wikimedia.org/wiki/Research:Understanding_Wikidata_Queries need quite a bit since we process every entry in wdqs_extract and I was wondering how many cores and how much memory we can use without conflicting with anyone else.
Greetings,
Adrian
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing listAnalytics@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics