Hello all,
soon the partion where /home is stored will be full (only 5GB are left). Below you will find the list of the Top-25-space-user. If you are on the list, please realy look why you need so much space and clean-up if possible (the normal quota for home is 500MB or 1GB). If there is no process until sunday (or if /home runs full), the roots will look into these homes and clean themself.
If you are not on the list, it doesn't hurt to have a look at your home too ;-).
Clean-up means: Delete or compress or truncate or move it to the user-store if other people can use it too (the user-store is no waste dump for big files no- one need of corse ;-)).
Sincerly DaB.
2.9G joegazz84 2.9G mzmcbride 3.0G danny_b 3.6G flacus 3.7G hydriz 3.9G rriver 4.8G saper 4.9G tparscal 5.1G cbm 6.8G werdna 7.8G mjbmr 8.0G hippietrail 8.4G voj 9.5G myst 11G kolossos 11G rsumi 12G wikitanvir 13G baptiste 17G dschwen 21G prolineserver 40G project 44G sk 70G daniel 90G grimlock 106G alebot
Hi,
"40G project" I think this should be OK ;-)
Best regards, Jan
-----Ursprüngliche Nachricht----- Von: toolserver-l-bounces@lists.wikimedia.org [mailto:toolserver-l-bounces@lists.wikimedia.org] Im Auftrag von DaB. Gesendet: Montag, 5. September 2011 16:10 An: toolserver-l@lists.wikimedia.org Betreff: [Toolserver-l] Clean-up your homes
Hello all,
soon the partion where /home is stored will be full (only 5GB are left). Below you will find the list of the Top-25-space-user. If you are on the list, please realy look why you need so much space and clean-up if possible (the normal quota for home is 500MB or 1GB). If there is no process until sunday (or if /home runs full), the roots will look into these homes and clean themself.
If you are not on the list, it doesn't hurt to have a look at your home too ;-).
Clean-up means: Delete or compress or truncate or move it to the user-store if other people can use it too (the user-store is no waste dump for big files no- one need of corse ;-)).
Sincerly DaB.
2.9G joegazz84 2.9G mzmcbride 3.0G danny_b 3.6G flacus 3.7G hydriz 3.9G rriver 4.8G saper 4.9G tparscal 5.1G cbm 6.8G werdna 7.8G mjbmr 8.0G hippietrail 8.4G voj 9.5G myst 11G kolossos 11G rsumi 12G wikitanvir 13G baptiste 17G dschwen 21G prolineserver 40G project 44G sk 70G daniel 90G grimlock 106G alebot
-- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885
< 4.8G saper
now:
238.8M saper
Btw. I have an automatically updated copy of Mediawiki SVN repository here, so you can save space if you have your own copy.
/mnt/user-store/mediawiki/
(the copy is being pulled via hgsvn into Mercurial, so Mercurial commands operate here on revisions - see hginit.com for introduction).
//Marcin
Marcin Cieslak wrote:
< 4.8G saper
now:
238.8M saper
Btw. I have an automatically updated copy of Mediawiki SVN repository here, so you can save space if you have your own copy.
/mnt/user-store/mediawiki/
(the copy is being pulled via hgsvn into Mercurial, so Mercurial commands operate here on revisions - see hginit.com for introduction).
//Marcin
Checkouts (often outdated) of mediawiki and pywikipedia are quite common. Maybe they should be provided as a toolserver service.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
...but I need exact the outdated pywikipedia checkout... ;))
(ok in fact it's a drtrigobot checkout which is based on pywikipedia)
Greetings DrTrigon
Am 05.09.2011 22:56, schrieb Platonides:
Marcin Cieslak wrote:
< 4.8G saper
now:
238.8M saper
Btw. I have an automatically updated copy of Mediawiki SVN repository here, so you can save space if you have your own copy.
/mnt/user-store/mediawiki/
(the copy is being pulled via hgsvn into Mercurial, so Mercurial commands operate here on revisions - see hginit.com for introduction).
//Marcin
Checkouts (often outdated) of mediawiki and pywikipedia are quite common. Maybe they should be provided as a toolserver service.
_______________________________________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
I'm the driver of Alebot, I apologyze for a mistake that wasted a lot of space; I misinterpreted the "1 giga limit" and I was confident that server would not allow that a larger server space is used. :-(. I was wrong.
Now, I deleted all what I could. Can anyone of you take a look and to tell me how much alebot is using now, if it's not a too annoying task? Thanks!
Alex brollo
On Tue, Sep 6, 2011 at 03:26, Alex Brollo alex.brollo@gmail.com wrote:
Can anyone of you take a look and to tell me how much alebot is using now, if it's not a too annoying task? Thanks! Alex brollo
Just run this yourself:
du -sh ~alebot
-Jeremy
Dr. Trigon dr.trigon@surfeu.ch wrote:
...but I need exact the outdated pywikipedia checkout... ;))
(ok in fact it's a drtrigobot checkout which is based on pywikipedia)
I looked at your SVN (in case we'd like to create a shared repository or put it into Pywiki's SVN) and I, too, found that you have actually your own copy of pywikipedia there. Are there any local changes that would need to be included in the mainline?
By the way, I and couple of other bot operators from plwiki decided to share our code in a single repository. We are using mercurial for that and it's publicly available:
http://bitbucket.org/plwiki/bot/
Pywikipedia-based code, Perl code and Common Lisp is available. Also some code for my wikipedia-related IRC bots is available there as well.
I never include pywikipedia snapshot in the repository, I have an SVN checkout in $HOME/wikipieda/pywikipedia and it's included in $PYTHONPATH so all imports work.
(If somebody would like to store their code in the above repository, just email me privately with your bitbucket ID).
//Saper
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I looked at your SVN (in case we'd like to create a shared repository or put it into Pywiki's SVN) and I, too, found that you have actually your own copy of pywikipedia there. Are there any local changes that would need to be included in the mainline?
There are additional files, according to my repo [1] and it is based on pywikipedia r8990 (not the newest one - which is important).
[1] https://fisheye.toolserver.org/changelog/drtrigon
So there are no changes to the files in r8990.
I never include pywikipedia snapshot in the repository, I have an SVN checkout in $HOME/wikipieda/pywikipedia and it's included in $PYTHONPATH so all imports work.
Is also an idea to consider. I have the snapshot include since it is needed in order for the bot to work - and crutial to use to correct revision. Thus I decided to include it to help other users to get started! ;) What I am trying now is to use externals in svn, but this does no change to the issues discussed here.
Greetings DrTrigon
Dr. Trigon dr.trigon@surfeu.ch wrote:
There are additional files, according to my repo [1] and it is based on pywikipedia r8990 (not the newest one - which is important).
[1] https://fisheye.toolserver.org/changelog/drtrigon
So there are no changes to the files in r8990.
is needed in order for the bot to work - and crutial to use to correct revision.
I extracted all files that are not there (there are also some i18n differences).
bot_control.py clean_user_sandbox.py Doxyfile dtbext dtbext/dtbext_pywikibot dtbext/dtbext_pywikibot/dtbext_exceptions.py dtbext/dtbext_pywikibot/__init__.py dtbext/dtbext_pywikibot/dtbext_textlib.py dtbext/dtbext_pywikibot/__init__.pyc dtbext/dtbext_basic.py dtbext/__init__.py dtbext/dtbext_userlib.py dtbext/dtbext_wikipedia.py dtbext/dtbext_date.py script_wui.py subster_irc.py subster.py sum_disc.py tests tests/dtbext_tester.py
Maybe the quick redirect code should be checked (I thing Page() does not need to full get() to check for redirect), maybe there should be a separate Section class in pywikipedia - are there are more issues?
Btw. did you try to make your WUI code to work as the userinterface (in "userinterfaces" directory)?
//Saper
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I extracted all files that are not there (there are also some i18n differences).
bot_control.py clean_user_sandbox.py Doxyfile dtbext dtbext/dtbext_pywikibot dtbext/dtbext_pywikibot/dtbext_exceptions.py dtbext/dtbext_pywikibot/__init__.py dtbext/dtbext_pywikibot/dtbext_textlib.py dtbext/dtbext_pywikibot/__init__.pyc dtbext/dtbext_basic.py dtbext/__init__.py dtbext/dtbext_userlib.py dtbext/dtbext_wikipedia.py dtbext/dtbext_date.py script_wui.py subster_irc.py subster.py sum_disc.py tests tests/dtbext_tester.py
Maybe the quick redirect code should be checked (I thing Page() does not need to full get() to check for redirect), maybe there should be a separate Section class in pywikipedia - are there are more issues?
Which "quick redirect code"? What are you doing? Do you want to commit the code upstream? There may be a lot of issues, since I had no time (at all ;) to have a look at such things recently... (e.g. 'i18n' might need special handling or some code changes...)
Btw. did you try to make your WUI code to work as the userinterface (in "userinterfaces" directory)?
WUI code is at the moment more a concept than a full running code. Some things still have to be solved - as you might see from looking at the VERY short and simple code... ;) I would have to consider this - but I not sure if the 'userinterface' of the framework fits...
Can you explain more clearly what you are doing at the moment? Please DO NOT CHANGE any code in my home!! :) (at least until we solved all issues!!)
Greetings DrTrigon
Dr. Trigon dr.trigon@surfeu.ch wrote:
Can you explain more clearly what you are doing at the moment? Please DO NOT CHANGE any code in my home!! :) (at least until we solved all issues!!)
I have checked out your code out of SVN and I am playing with it. I don't think I have commit rights to your repository, so you shouldn't worry.
I was just interested to know what are the issues with running your code with the latest pywikipedia trunk and I just wanted to figure out why.
//Saper
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I have checked out your code out of SVN and I am playing with it. I don't think I have commit rights to your repository, so you shouldn't worry.
I'm not worried about the repo, but my home dir on the TS... since I am FULLY trusting all TS-users... ;)
So you was always referring to the repos instead of the pywikipedia framework copies on the TS drives?
I was just interested to know what are the issues with running your code with the latest pywikipedia trunk and I just wanted to figure out why.
Aaaaa! ;))) Now that is good to hear! Please keep me up2date! :)) In fact I do not know what issues are there and that's the reason why I do not automatically update or use the newest framework...
Instead I frequently update DrTrigonBot to the newest pywikipedia framework manually (merge) and then do a lot of testing, comparing and adapting to get a new nice (and working) package... :))
Greetings
Hello again,
short progress report: The situation was better arround Wednesday, but now the situation is bad again (not as bad as at Monday, but still bad). You here is the new who-uses-the-most-space-list:
3.0G dispenser 3.1G danny_b 3.6G flacus 3.7G hydriz 3.9G rriver 4.9G tparscal 5.1G cbm 6.8G werdna 7.8G mjbmr 8.0G hippietrail 8.4G voj 8.7G kolossos 9.6G myst 11G rsumi 12G wikitanvir 13G baptiste 17G dschwen 44G sk 61G daniel 92G grimlock
The projects-directory increased by 100GB (that's the lion-share of the "bad"). So please if you run a MMP, please look at it too.
Sincerly, DaB.
Alebot and it's horrible 100Gby space wasting has disappeared from the list, I'm happy from this but it is all what I could do. Isn't it possible to block user space at 1Gby? I'd like much more to be blocked in my user space, tnah to cause another time so big troubles - I guess other users do too.
Alex brollo
2011/9/9 DaB. WP@daniel.baur4.info
Hello again,
short progress report: The situation was better arround Wednesday, but now the situation is bad again (not as bad as at Monday, but still bad). You here is the new who-uses-the-most-space-list:
3.0G dispenser 3.1G danny_b 3.6G flacus 3.7G hydriz 3.9G rriver 4.9G tparscal 5.1G cbm 6.8G werdna 7.8G mjbmr 8.0G hippietrail 8.4G voj 8.7G kolossos 9.6G myst 11G rsumi 12G wikitanvir 13G baptiste 17G dschwen 44G sk 61G daniel 92G grimlock
The projects-directory increased by 100GB (that's the lion-share of the "bad"). So please if you run a MMP, please look at it too.
Sincerly, DaB.
-- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Alex Brollo alex.brollo@gmail.com wrote:
--===============9015774349093925025== Content-Type: multipart/alternative; boundary=001636416fa5e8017904ac989ba1
--001636416fa5e8017904ac989ba1 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Alebot and it's horrible 100Gby space wasting has disappeared from the list= , I'm happy from this but it is all what I could do. Isn't it possible to block user space at 1Gby? I'd like much more to be blocked in my user space, tnah to cause another time so big troubles - I guess other users do too.
Why be blocked? Why there was so much space? If there were logs, maybe we should put simple instructions for bot operators how to trim them?
//Saper
Hello,
I (sk) have 44G and I am not happy about that. But when I scan the dumps from many languages for the Templatetiger, then I need a space for the results. At the moment I download and unzip the XML-Dumps under /mnt/user-store/dump. The txt-files of the result are stored in my home-directory. This will take alone 39G. User:Kolossos need the results of the scan for the next step in the templatetiger-process.
It is a way to store the results under /mnt/user-store/sk ?
Stefan (sk)
Am 10.09.2011 18:15, schrieb Alex Brollo:
Alebot and it's horrible 100Gby space wasting has disappeared from the list, I'm happy from this but it is all what I could do. Isn't it possible to block user space at 1Gby? I'd like much more to be blocked in my user space, tnah to cause another time so big troubles - I guess other users do too.
Alex brollo
2011/9/9 DaB. <WP@daniel.baur4.info mailto:WP@daniel.baur4.info>
Hello again, short progress report: The situation was better arround Wednesday, but now the situation is bad again (not as bad as at Monday, but still bad). You here is the new who-uses-the-most-space-list: 3.0G dispenser 3.1G danny_b 3.6G flacus 3.7G hydriz 3.9G rriver 4.9G tparscal 5.1G cbm 6.8G werdna 7.8G mjbmr 8.0G hippietrail 8.4G voj 8.7G kolossos 9.6G myst 11G rsumi 12G wikitanvir 13G baptiste 17G dschwen 44G sk 61G daniel 92G grimlock The projects-directory increased by 100GB (that's the lion-share of the "bad"). So please if you run a MMP, please look at it too. Sincerly, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885 _______________________________________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org <mailto:Toolserver-l@lists.wikimedia.org>) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Stefan Kühn kuehn-s@gmx.net wrote:
Hello,
I (sk) have 44G and I am not happy about that. But when I scan the dumps from many languages for the Templatetiger, then I need a space for the results. At the moment I download and unzip the XML-Dumps under /mnt/user-store/dump. The txt-files of the result are stored in my home-directory. This will take alone 39G. User:Kolossos need the results of the scan for the next step in the templatetiger-process.
Maybe there is a way to work on compressed dumps?
//Saper
Stefan Kühn wrote:
I (sk) have 44G and I am not happy about that. But when I scan the dumps from many languages for the Templatetiger, then I need a space for the results. At the moment I download and unzip the XML-Dumps under /mnt/user-store/dump. The txt-files of the result are stored in my home-directory. This will take alone 39G. User:Kolossos need the results of the scan for the next step in the templatetiger-process.
It is a way to store the results under /mnt/user-store/sk ?
I didn't really understand this post. Just tell the script to output the results to /mnt/user-store/sk/results.txt. What's the problem?
MZMcBride
Stefan Kühn wrote:
I (sk) have 44G and I am not happy about that. But when I scan the dumps from many languages for the Templatetiger, then I need a space for the results. At the moment I download and unzip the XML-Dumps under /mnt/user-store/dump. The txt-files of the result are stored in my home-directory. This will take alone 39G. User:Kolossos need the results of the scan for the next step in the templatetiger-process.
It is a way to store the results under /mnt/user-store/sk ?
I didn't really understand this post. Just tell the script to output the results to /mnt/user-store/sk/results.txt. What's the problem?
What is allowed to store in this directory /mnt/user-store/ ?
I don't want crash this directory with my data. Why it is a problem to store big datasets in my home-directory and not a problem to store it under /mnt/user-store/ ?
Yes it is no problem to change the savepoint in my script, but it this what we want? IMHO than we don't need a home-directory. :-) Also no skript can check how much data every user store at the toolserver.
Stefan (sk)
Hello, At Thursday 15 September 2011 14:14:36 DaB. wrote:
What is allowed to store in this directory /mnt/user-store/ ?
As long as other users can/will use the data too: yes.
I don't want crash this directory with my data. Why it is a problem to store big datasets in my home-directory and not a problem to store it under /mnt/user-store/ ?
There are 2 main differences between home and user-store: -user-store have way more space then home -user-store is not back-uped.
It is no problem to store big files in your home as long as you need them.
Sincerly, DaB.
Hello, At Thursday 15 September 2011 14:14:36 DaB. wrote:
What is allowed to store in this directory /mnt/user-store/ ?
As long as other users can/will use the data too: yes.
I don't want crash this directory with my data. Why it is a problem to store big datasets in my home-directory and not a problem to store it under /mnt/user-store/ ?
There are 2 main differences between home and user-store: -user-store have way more space then home -user-store is not back-uped.
It is no problem to store big files in your home as long as you need them.
Thanks, for this info. I don't need a backup of this data, so I will change the savepoint of the data in my script.
Stefan
DaB. wrote:
There are 2 main differences between home and user-store: -user-store have way more space then home -user-store is not back-uped.
It is no problem to store big files in your home as long as you need them.
/home/<user>/ has a space quota and requires bothering sysadmins to increase the quota as necessary. /mnt/user-store/ has over 800 GB available and has no quotas.
Stefan Kühn wrote:
Thanks, for this info. I don't need a backup of this data, so I will change the savepoint of the data in my script.
Sounds like a good idea. :-)
MZMcBride
toolserver-l@lists.wikimedia.org