-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
If you are currently using disk space on user-store, please check your usage and delete any unneeded files. There is only 70GB of space left on the volume now.
(I will produce a more detailed report of who's using space later...)
- river.
River Tarnell wrote:
If you are currently using disk space on user-store, please check your usage and delete any unneeded files. There is only 70GB of space left on the volume now.
(I will produce a more detailed report of who's using space later...)
I can save you some time:
$ du -h /mnt/user-store/stats 822G /mnt/user-store/stats
==> over 20% of the data is mine...
This contains the raw data for the consultation statistics for all projects. It can easily be cleaned, since there is data starting in January 2009 in there. But I'd be glad to know who uses it, and if people only use the most recent files, or if some of them use the archives -- so that I know what can be cleaned (it could even be done automatically).
I remember there were discussions about having some dedicated space for these files (and a more accessible version of them), I don't know if this went anywhere ?
Just in case, I keep a copy of all older files, and can provide them to anyone who would need them. In the recent past, I did not want to remove them, because there were not too many copies of them around, and I was glad to have a copy on the toolserver, but the WMF has now downloaded a copy of them, so I don't mind so much deleting them
Frédéric
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Frederic Schutz:
==> over 20% of the data is mine...
This is only 20%, and as far as I know does not grow at the rate that would explain the recent increase in user-store disk usage. Still, if you find no one is using the older files, and they are available somewhere else, they could be deleted.
I remember there were discussions about having some dedicated space for these files (and a more accessible version of them), I don't know if this went anywhere ?
We are working on this at the moment.
- river.
On 9/7/2010 10:11 AM, Frederic Schutz wrote:
River Tarnell wrote:
If you are currently using disk space on user-store, please check your usage and delete any unneeded files. There is only 70GB of space left on the volume now.
(I will produce a more detailed report of who's using space later...)
I can save you some time:
$ du -h /mnt/user-store/stats 822G /mnt/user-store/stats
==> over 20% of the data is mine...
This contains the raw data for the consultation statistics for all projects. It can easily be cleaned, since there is data starting in January 2009 in there. But I'd be glad to know who uses it, and if people only use the most recent files, or if some of them use the archives -- so that I know what can be cleaned (it could even be done automatically).
I use these, mainly only the newest ones. Though I'll need all of January 2010 rather soon.
I remember there were discussions about having some dedicated space for these files (and a more accessible version of them), I don't know if this went anywhere ?
Just in case, I keep a copy of all older files, and can provide them to anyone who would need them. In the recent past, I did not want to remove them, because there were not too many copies of them around, and I was glad to have a copy on the toolserver, but the WMF has now downloaded a copy of them, so I don't mind so much deleting them
Are these publicly accessible somewhere? Domas's site only keeps the last few months - http://dammit.lt/wikistats/archive/2010/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Here is the per-user usage report:
schutz : 822.69G in 14604 files cmarqu : 492.62G in 126629282 files multichill : 475.83G in 2261720 files dschwen : 433.88G in 16212733 files sk : 169.82G in 83 files emijrp : 82.75G in 583 files cbm : 49.34G in 10035 files bryan : 19.57G in 1162796 files drh08 : 17.33G in 1837 files dispenser : 15.03G in 2 files johang : 12.09G in 2404 files mzmcbride : 11.66G in 2 files hippietrail : 7.55G in 30 files seb35 : 7.00G in 28 files enwp10 : 6.09G in 17 files fale : 5.32G in 1 files contests : 3.11G in 1293 files holek : 2.96G in 1 files pathoschild : 2.61G in 1 files saper : 2.51G in 112507 files liangent : 2.44G in 1 files autocracy : 2.12G in 1 files valhallasw : 1.81G in 1 files enwikt : 1.49G in 5 files danny_b : 1.34G in 19 files daniel : 489.88M in 1 files magnus : 430.12M in 1150 files vyznev : 269.83M in 1 files balu : 206.10M in 2 files mauro742 : 38.48M in 1 files overlordq : 3.19M in 1 files
- river.
I've freed my 5.some gbs, since I don't use it any more
On Wed, Sep 8, 2010 at 12:58 PM, River Tarnell river.tarnell@wikimedia.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Here is the per-user usage report:
schutz : 822.69G in 14604 files cmarqu : 492.62G in 126629282 files multichill : 475.83G in 2261720 files dschwen : 433.88G in 16212733 files sk : 169.82G in 83 files emijrp : 82.75G in 583 files cbm : 49.34G in 10035 files bryan : 19.57G in 1162796 files drh08 : 17.33G in 1837 files dispenser : 15.03G in 2 files johang : 12.09G in 2404 files mzmcbride : 11.66G in 2 files hippietrail : 7.55G in 30 files seb35 : 7.00G in 28 files enwp10 : 6.09G in 17 files fale : 5.32G in 1 files contests : 3.11G in 1293 files holek : 2.96G in 1 files pathoschild : 2.61G in 1 files saper : 2.51G in 112507 files liangent : 2.44G in 1 files autocracy : 2.12G in 1 files valhallasw : 1.81G in 1 files enwikt : 1.49G in 5 files danny_b : 1.34G in 19 files daniel : 489.88M in 1 files magnus : 430.12M in 1150 files vyznev : 269.83M in 1 files balu : 206.10M in 2 files mauro742 : 38.48M in 1 files overlordq : 3.19M in 1 files
- river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD)
iEYEARECAAYFAkyHbEsACgkQIXd7fCuc5vLisACeO1hmNfXxIF9fSywBWuzhHyLI 5dgAn3C4v9a9vItsIfzG8oMa5Sq5H2ok =Jrtq -----END PGP SIGNATURE-----
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Daniel Schwen:
Uhm, I see 215G in /mnt/user-store/wikiminiatlas/, I thought that is the lions share of my stuff. Where is the other half?
hemlock# du -sh iip-cache-dschwen 201G iip-cache-dschwen
- river.
hemlock# du -sh iip-cache-dschwen 201G iip-cache-dschwen
Oh, right, I forgot about that (checked it yesterday and thought in relation to the 3.7TB it was negligible...). That directory contains multiresolution images generated from very large images on commons for the zoomviewer [1]. Generating them takes up to more than a minute each, and many of them actually had to be generated on an external server (as the process uses a lot of memory for a short time and is frequently killed by slayerd). Deleting them is not really an option (well it is, but it would mean discontinuing the service). Daniel
[1] http://toolserver.org/~dschwen/iip/wip.php?f=Seattle_7.jpg
2010/9/8 River Tarnell river.tarnell@wikimedia.de:
Here is the per-user usage report:
cmarqu : 492.62G in 126629282 files
These are the hillshading tiles used in e.g. http://hikebikemap.de (and some other sites as well now). I had made a rather unsuccessful attempt in re-packing them into metatiles in the past, which I could try another time if the number of files became a problem (which you said wasn't, back then).
Cheers Colin
Am 09.09.2010 22:34, schrieb Colin Marquardt:
These are the hillshading tiles used in e.g. http://hikebikemap.de
It would be very sad if we would need to delete them.
Colin, how completed is the rendering of the hillshading tiles? I remember that it was still running on cassini when it got re-purposed.
Peter
2010/9/9 Peter Körner osm-lists@mazdermind.de:
Am 09.09.2010 22:34, schrieb Colin Marquardt:
These are the hillshading tiles used in e.g. http://hikebikemap.de
It would be very sad if we would need to delete them.
AIUI, River was just asking who *could* free space, not pushing anybody to do so for useful stuff.
Colin, how completed is the rendering of the hillshading tiles? I remember that it was still running on cassini when it got re-purposed.
I don't remember exactly, I got Europe and a part of Asia down to zoom 15, with parts down to 16 I think, and North America to zoom 14 or some such. I always wanted to make the rendering dynamic (and then never expire the tiles), but it would need a post-processing task in either renderd or tirex (the latter seems a bit more suited for that from what I read) for the alpha-channel stuff, and ideally do the metatiling in the same step.
Cheers Colin
Am 09.09.2010 23:04, schrieb Colin Marquardt:
AIUI, River was just asking who *could* free space, not pushing anybody to do so for useful stuff.
I just wanted to mention that there's a need for them.
I always wanted to make the rendering dynamic (and then never expire the tiles), but it would need a post-processing task in either renderd or tirex (the latter seems a bit more suited for that from what I read) for the alpha-channel stuff, and ideally do the metatiling in the same step.
It should not be to hard to write your own renderer. You may want to take a look at
http://trac.openstreetmap.org/browser/applications/utils/tirex/lib/Tirex/Backend/Test.pm and http://trac.openstreetmap.org/browser/applications/utils/tirex/backends/test
Peter
2010/9/9 Peter Körner osm-lists@mazdermind.de:
Am 09.09.2010 23:04, schrieb Colin Marquardt:
I always wanted to make the rendering dynamic (and then never expire the tiles), but it would need a post-processing task in either renderd or tirex (the latter seems a bit more suited for that from what I read) for the alpha-channel stuff, and ideally do the metatiling in the same step.
It should not be to hard to write your own renderer. You may want to take a look at
http://trac.openstreetmap.org/browser/applications/utils/tirex/lib/Tirex/Backend/Test.pm and http://trac.openstreetmap.org/browser/applications/utils/tirex/backends/test
Thanks, that sounds doable. As winter draws nearer, time at home will increase :)
Cheers Colin
Am 09.09.2010, 23:28 Uhr, schrieb Peter Körner osm-lists@mazdermind.de:
Am 09.09.2010 23:04, schrieb Colin Marquardt:
I always wanted to make the rendering dynamic (and then never expire the tiles)
It should not be to hard to write your own renderer.
I wrote a WMS server for this, it's easy.
But I have the problems that with mod_tile+tirex, my tiles always expire, and I don't know why. I've set all timeouts to dozens of weeks, but they keep expiring after 1 day or so.
It MIGHT have something to do that I purge and fill my database daily, and so it assumes that the data changed, and triggers a dirty state, but I found neigher indication of this, nor documentation, nor log entries, debugging possibilities or similar.
So I consider switching back to tilecache, sadly.
Kind regards, Kay
Am 10.09.2010 11:21, schrieb Kay Drangmeister:
Am 09.09.2010, 23:28 Uhr, schrieb Peter Körnerosm-lists@mazdermind.de:
Am 09.09.2010 23:04, schrieb Colin Marquardt:
I always wanted to make the rendering dynamic (and then never expire the tiles)
It should not be to hard to write your own renderer.
I wrote a WMS server for this, it's easy.
But I have the problems that with mod_tile+tirex, my tiles always expire, and I don't know why. I've set all timeouts to dozens of weeks, but they keep expiring after 1 day or so.
As far as I understood the process, the expiry is done by mod_tile [1]. You might need to patch that out somehow.
It it really is mod_tile, then you should see messages in your apache log when you set the apache log-level [2] to debug.
Peter
[1] http://trac.openstreetmap.org/browser/applications/utils/mod_tile/mod_tile.c#L276 [2] http://httpd.apache.org/docs/2.0/mod/core.html#loglevel
toolserver-l@lists.wikimedia.org