Hi,
I have installed Maria DB to all my servers, including production servers few weeks ago, and I found it quite stable and I like it (even the command line tool for working with sql is far better than the one included in mysql pack)
It's supported on all latest ubuntu versions from 10.04 UP (maybe even older) - so my question is, are we going to use it on wikimedia production?
I think we could migrate beta cluster for now - it has terrible performance and this could help. It could be a first step to migrate wikimedia production cluster.
Umm there was a thread several months ago about how it is used on several of the slave dbs, if I recall.
-bawolff On 2013-02-13 8:28 AM, "Petr Bena" benapetr@gmail.com wrote:
Hi,
I have installed Maria DB to all my servers, including production servers few weeks ago, and I found it quite stable and I like it (even the command line tool for working with sql is far better than the one included in mysql pack)
It's supported on all latest ubuntu versions from 10.04 UP (maybe even older) - so my question is, are we going to use it on wikimedia production?
I think we could migrate beta cluster for now - it has terrible performance and this could help. It could be a first step to migrate wikimedia production cluster. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Feb 13, 2013 at 8:05 AM, bawolff bawolff+wn@gmail.com wrote:
Umm there was a thread several months ago about how it is used on several of the slave dbs, if I recall.
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Okay - so what is outcome? Should we migrate beta cluster? Are we going to use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad innocentkiller@gmail.com wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff bawolff+wn@gmail.com wrote:
Umm there was a thread several months ago about how it is used on several of the slave dbs, if I recall.
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Feb 13, 2013 at 6:19 AM, Petr Bena benapetr@gmail.com wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we going to use it in production?
At the risk of derailing the conversation to an unrelated subject, I would rather work on finding a way to keep the db on beta cluster up to date rather than migrate to a whole different SQL implementation that is still not correct. https://bugzilla.wikimedia.org/show_bug.cgi?id=36228
-Chris
As far as I know we're using a custom mariadb version in production, not the ubuntu version. It would be ideal to use the same versions for everything, but just switching out to MariaDB isn't going to solve anything for us.
Also, we're not using MariaDB everywhere in production. We're using a mix of MySQL and MariaDB. Let's not change anything till we know how that's going to work out in production.
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena benapetr@gmail.com wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we going to use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad innocentkiller@gmail.com wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff bawolff+wn@gmail.com wrote:
Umm there was a thread several months ago about how it is used on
several
of the slave dbs, if I recall.
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up my time, but the trial production roll-out will resume this month. All signs still point to our using it in production.
I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about query optimizer changes making one particular query type significantly slower. In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first production slave switch.
Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types in 5.5, breaking replication and requiring one off rewriting and execution of the query locally to ensure data consistency before skipping over it. I opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki.
That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to be a slow and methodical process at this stage. Bugs in extensions that aren't used by English Wikipedia may be found and require fixing along the way.
As the MariaDB roll-out proceeds, I will provide updates on wikitech-l.
Best, Asher
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena benapetr@gmail.com wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we going to use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad innocentkiller@gmail.com wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff bawolff+wn@gmail.com wrote:
Umm there was a thread several months ago about how it is used on
several
of the slave dbs, if I recall.
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
thanks for updates.
Can you tell me what is a difference between maria db you are using and the version that is recommended for use on ubuntu?
On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman afeldman@wikimedia.orgwrote:
The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up my time, but the trial production roll-out will resume this month. All signs still point to our using it in production.
I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about query optimizer changes making one particular query type significantly slower. In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first production slave switch.
Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types in 5.5, breaking replication and requiring one off rewriting and execution of the query locally to ensure data consistency before skipping over it. I opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki.
That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to be a slow and methodical process at this stage. Bugs in extensions that aren't used by English Wikipedia may be found and require fixing along the way.
As the MariaDB roll-out proceeds, I will provide updates on wikitech-l.
Best, Asher
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena benapetr@gmail.com wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we going
to
use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad innocentkiller@gmail.com wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff bawolff+wn@gmail.com wrote:
Umm there was a thread several months ago about how it is used on
several
of the slave dbs, if I recall.
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
For most projects, I recommend using the official packages available via the MariaDB projects own apt repo.
The official packages are based on the Debian mysql packaging where installing the server package also installs a default database created around generic config defaults, a debian mysql maintenance user with a randomly generated password, and scripts (including init) that assume privileged access via that user. That is, installing the packages provides you with a fresh running working database with generic defaults suitable for a small server, and certain admin tasks automated. I think that's what the average labs and general users wants and expects.
The packages I've built for production use at wmf strips out all of the debianisms, the debian project script rewrites, the pre/post install actions. They also leave debug symbols in the binaries and have compiler flag tweaks, but do not at this stage contain any source patches. Installing the server package doesn't create a default db, or provide an environment where you can even start the server on a fresh sever install without further work. Probably not a good choice for most labs users.
On Wednesday, February 13, 2013, Petr Bena wrote:
thanks for updates.
Can you tell me what is a difference between maria db you are using and the version that is recommended for use on ubuntu?
On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman <afeldman@wikimedia.org<javascript:_e({}, 'cvml', 'afeldman@wikimedia.org');>
wrote:
The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up my time, but the trial production roll-out will resume this month. All signs still point to our using it in production.
I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about query optimizer changes making one particular query type significantly slower. In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first production slave switch.
Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types in 5.5, breaking replication and requiring one off rewriting and execution of the query locally to ensure data consistency before skipping over it. I opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki.
That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to be a slow and methodical process at this stage. Bugs in extensions that aren't used by English Wikipedia may be found and require fixing along the way.
As the MariaDB roll-out proceeds, I will provide updates on wikitech-l.
Best, Asher
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena <benapetr@gmail.com<javascript:_e({}, 'cvml', 'benapetr@gmail.com');>> wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we going
to
use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad <innocentkiller@gmail.com<javascript:_e({}, 'cvml', 'innocentkiller@gmail.com');>>
wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff <bawolff+wn@gmail.com<javascript:_e({}, 'cvml', 'bawolff%2Bwn@gmail.com');>>
wrote:
Umm there was a thread several months ago about how it is used on
several
of the slave dbs, if I recall.
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');>
Labs-l mailing list Labs-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Labs-l@lists.wikimedia.org');>
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml', 'Wikitech-l@lists.wikimedia.org');> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Keeping debug symbols in binaries will result in poor performance, or it should
On Thu, Feb 14, 2013 at 4:47 PM, Asher Feldman afeldman@wikimedia.orgwrote:
For most projects, I recommend using the official packages available via the MariaDB projects own apt repo.
The official packages are based on the Debian mysql packaging where installing the server package also installs a default database created around generic config defaults, a debian mysql maintenance user with a randomly generated password, and scripts (including init) that assume privileged access via that user. That is, installing the packages provides you with a fresh running working database with generic defaults suitable for a small server, and certain admin tasks automated. I think that's what the average labs and general users wants and expects.
The packages I've built for production use at wmf strips out all of the debianisms, the debian project script rewrites, the pre/post install actions. They also leave debug symbols in the binaries and have compiler flag tweaks, but do not at this stage contain any source patches. Installing the server package doesn't create a default db, or provide an environment where you can even start the server on a fresh sever install without further work. Probably not a good choice for most labs users.
On Wednesday, February 13, 2013, Petr Bena wrote:
thanks for updates.
Can you tell me what is a difference between maria db you are using and the version that is recommended for use on ubuntu?
On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman <afeldman@wikimedia.org<javascript:_e({},
'cvml', 'afeldman@wikimedia.org');>
wrote:
The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up my time, but the trial production roll-out will resume this month. All
signs
still point to our using it in production.
I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about
query
optimizer changes making one particular query type significantly slower. In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first
production
slave switch.
Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types
in
5.5, breaking replication and requiring one off rewriting and execution
of
the query locally to ensure data consistency before skipping over it. I opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki.
That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to
be
a slow and methodical process at this stage. Bugs in extensions that
aren't
used by English Wikipedia may be found and require fixing along the way.
As the MariaDB roll-out proceeds, I will provide updates on wikitech-l.
Best, Asher
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena <benapetr@gmail.com<javascript:_e({},
'cvml', 'benapetr@gmail.com');>>
wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we
going
to
use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad <innocentkiller@gmail.com<javascript:_e({},
'cvml', 'innocentkiller@gmail.com');>>
wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff <bawolff+wn@gmail.com<javascript:_e({},
'cvml', 'bawolff%2Bwn@gmail.com');>>
wrote:
Umm there was a thread several months ago about how it is used on
several
of the slave dbs, if I recall.
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');>
Labs-l mailing list Labs-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Labs-l@lists.wikimedia.org');>
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml', 'Wikitech-l@lists.wikimedia.org');> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Er, no it shouldn't. Initial execution might take microseconds longer due to larger binary sizes and the elf loader having to skip over the symbols but that's about it.
On Thursday, February 14, 2013, Petr Bena wrote:
Keeping debug symbols in binaries will result in poor performance, or it should
On Thu, Feb 14, 2013 at 4:47 PM, Asher Feldman <afeldman@wikimedia.org<javascript:_e({}, 'cvml', 'afeldman@wikimedia.org');>
wrote:
For most projects, I recommend using the official packages available via the MariaDB projects own apt repo.
The official packages are based on the Debian mysql packaging where installing the server package also installs a default database created around generic config defaults, a debian mysql maintenance user with a randomly generated password, and scripts (including init) that assume privileged access via that user. That is, installing the packages provides you with a fresh running working database with generic defaults suitable for a small server, and certain admin tasks automated. I think that's what the average labs and general users wants and expects.
The packages I've built for production use at wmf strips out all of the debianisms, the debian project script rewrites, the pre/post install actions. They also leave debug symbols in the binaries and have compiler flag tweaks, but do not at this stage contain any source patches. Installing the server package doesn't create a default db, or provide an environment where you can even start the server on a fresh sever install without further work. Probably not a good choice for most labs users.
On Wednesday, February 13, 2013, Petr Bena wrote:
thanks for updates.
Can you tell me what is a difference between maria db you are using and the version that is recommended for use on ubuntu?
On Wed, Feb 13, 2013 at 6:58 PM, Asher Feldman <afeldman@wikimedia.org<javascript:_e({}, 'cvml', 'afeldman@wikimedia.org');><javascript:_e({},
'cvml', 'afeldman@wikimedia.org <javascript:_e({}, 'cvml', 'afeldman@wikimedia.org');>');>
wrote:
The production migration to MariaDB was paused for a time by the EQIAD datacenter migration and issues involving other projects that took up
my
time, but the trial production roll-out will resume this month. All
signs
still point to our using it in production.
I did a lot of query testing on an enwiki MariaDB 5.5 slave over the course of more than a month before the first production deployment. Major version migrations with mysql and derivatives are not to be taken lightly in production environments. At a minimum, one must be concerned about
query
optimizer changes making one particular query type significantly
slower.
In the case of the switch to 5.5, there are several default behavior changes over 5.1 that can break applications or change results. Hence, some serious work over a plodding time frame before that first
production
slave switch.
Despite those efforts, a couple weeks after the switch, I saw a query generated by what seems to be a very rare edge case from that AFTv4 extension that violated stricter enforcement of unsigned integer types
in
5.5, breaking replication and requiring one off rewriting and
execution of
the query locally to ensure data consistency before skipping over it.
I
opened a bug, Mathias fixed the extension, and I haven't seen any other compatibility issues from AFTv4 or anything else deployed on enwiki.
That said, other projects utilize different extensions, so all of my testing that has gone into enwiki cannot be assumed to fully cover everything else. Because of that, and because I want to continue proceeding with caution for all of our projects, this will continue to
be
a slow and methodical process at this stage. Bugs in extensions that
aren't
used by English Wikipedia may be found and require fixing along the
way.
As the MariaDB roll-out proceeds, I will provide updates on wikitech-l.
Best, Asher
On Wed, Feb 13, 2013 at 5:19 AM, Petr Bena <benapetr@gmail.com<javascript:_e({}, 'cvml', 'benapetr@gmail.com');><javascript:_e({},
'cvml', 'benapetr@gmail.com <javascript:_e({}, 'cvml', 'benapetr@gmail.com');>');>>
wrote:
Okay - so what is outcome? Should we migrate beta cluster? Are we
going
to
use it in production?
On Wed, Feb 13, 2013 at 2:08 PM, Chad <innocentkiller@gmail.com<javascript:_e({}, 'cvml', 'innocentkiller@gmail.com');><javascript:_e({},
'cvml', 'innocentkiller@gmail.com <javascript:_e({}, 'cvml', 'innocentkiller@gmail.com');>');>>
wrote:
On Wed, Feb 13, 2013 at 8:05 AM, bawolff <bawolff+wn@gmail.com<javascript:_e({}, 'cvml', 'bawolff%2Bwn@gmail.com');><javascript:_e({},
'cvml', 'bawolff%2Bwn@gmail.com <javascript:_e({}, 'cvml', 'bawolff%252Bwn@gmail.com');>');>>
wrote:
> Umm there was a thread several months ago about how it is used on several > of the slave dbs, if I recall. >
Indeed, you're looking for "mariadb 5.5 in production for english wikipedia"
http://www.gossamer-threads.com/lists/wiki/wikitech/319925
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');> <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');>');>
Labs-l mailing list Labs-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Labs-l@lists.wikimedia.org');> <javascript:_e({}, 'cvml',
'Labs-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Labs-l@lists.wikimedia.org');>');>
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');> <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml',
'Wikitech-l@lists.wikimedia.org');>');>
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org <javascript:_e({}, 'cvml', 'Wikitech-l@lists.wikimedia.org');> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Feb 14, 2013, at 5:02 PM, Petr Bena benapetr@gmail.com wrote:
Keeping debug symbols in binaries will result in poor performance, or it should
That's bollocks. It results in a larger binary _on disk_. The symbol table isn't even loaded into memory and doesn't affect performance.
Debug information is *highly useful* in a production setup, and we try to run all our core applications with it so we have a chance to debug issues when they occur.
I think the only reason distributions omit debug information is to save disk space.
On Thu, Feb 14, 2013 at 05:14:31PM +0100, Mark Bergsma wrote:
Debug information is *highly useful* in a production setup, and we try to run all our core applications with it so we have a chance to debug issues when they occur.
I think the only reason distributions omit debug information is to save disk space.
A lot of Debian packages ship -dbg alongside the main package, that contains the stripped-out debug symbols in /usr/lib/debug (gdb loads those automatically, either based on the filename or build-id). The toolchain handles this more or less automatically, but it still needs maintainer action to define this separate package and upload it with every package upload.
Ubuntu has experimented in the past with the concept of automatically generating and shipping symbols for *all* packages, packaged up in a "ddeb"s (same format as .deb) and shipped via a different repository that isn't mirrored by all of the downstream mirrors.
This was years ago, I'm not sure what has happened since then. I remember being discussed in Debian as well, but it was never adopted, probably because noone ever implemented it :)
For MySQL/MariaDB, it seems that the Debian packages don't ship a -dbg package by default. That's a shame, we can ask for that. As for the rest of Asher's changes, I'd love to find a way to make stock packages work in our production setup, but I'm not sure if the maintainer would welcome the extra complexity of conditionally switching behaviors. We can try if you're willing to, Asher :)
Regards, Faidon
I would much rather abandon using debs than use what the debian project has done to mysql packaging in any production environment. If the discussion has come down to this, I did WMF a disservice by drifting away from Domas' optimized "make ; make install ; rsync unstripped binaries to prod" workflow.
In general, I find environments that don't individually package according to distro standards every part of their core application stack that gets built in-house to be more productive, and more responsive to the needs of developers and ultimately the application. When an ops team claims that building a recent version of libmemcached for a stable OS is almost impossibly hard and will take weeks because it requires backporting a debian maintainers packaging of it for an experimental distro with that distros unrelated library version dependencies and reliance on a newer incompatible dpkg tool chain, there's probably something wrong with that workflow.
I like to rely on Linux distros for the lowest common denominator layer of the stack and related security updates. The approach that goes into building and maintaining such a beast are rather different than the concerns that go into operating a continually developed and deployed distributed application used by half a billion people. I don't see a win in trying to force the two together.
On Thursday, February 14, 2013, Faidon Liambotis wrote:
For MySQL/MariaDB, it seems that the Debian packages don't ship a -dbg package by default. That's a shame, we can ask for that. As for the rest of Asher's changes, I'd love to find a way to make stock packages work in our production setup, but I'm not sure if the maintainer would welcome the extra complexity of conditionally switching behaviors. We can try if you're willing to, Asher :)
Regards, Faidon
Labs-l mailing list Labs-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/labs-l
On 14/02/13 18:26, Faidon Liambotis wrote:
Ubuntu has experimented in the past with the concept of automatically generating and shipping symbols for *all* packages, packaged up in a "ddeb"s (same format as .deb) and shipped via a different repository that isn't mirrored by all of the downstream mirrors.
This was years ago, I'm not sure what has happened since then. I remember being discussed in Debian as well, but it was never adopted, probably because noone ever implemented it :)
Good question. There are a few bugs and blueprints about it, and they show as *implemented*
https://blueprints.launchpad.net/ubuntu/+spec/apt-get-debug-symbols https://bugs.launchpad.net/ubuntu/+bug/14484 https://lists.ubuntu.com/archives/ubuntu-devel-announce/2006-September/00019...
What happened, then?
On 02/14/2013 05:11 PM, Platonides wrote:
On 14/02/13 18:26, Faidon Liambotis wrote:
This was years ago, I'm not sure what has happened since then. I remember being discussed in Debian as well, but it was never adopted, probably because noone ever implemented it :)
Good question. There are a few bugs and blueprints about it, and they show as *implemented*
What happened, then?
It has been implemented on the Ubuntu side for a while. See https://wiki.ubuntu.com/DebuggingProgramCrash
I just switched back to Debian but haven't had the need to see what is going on with ddebs -- till now!
You can see the rationale here: http://wiki.debian.org/AutomaticDebugPackages
Packages are hosted on http://debug.debian.net/
On 15/02/13 09:11, Platonides wrote:
On 14/02/13 18:26, Faidon Liambotis wrote:
Ubuntu has experimented in the past with the concept of automatically generating and shipping symbols for *all* packages, packaged up in a "ddeb"s (same format as .deb) and shipped via a different repository that isn't mirrored by all of the downstream mirrors.
This was years ago, I'm not sure what has happened since then. I remember being discussed in Debian as well, but it was never adopted, probably because noone ever implemented it :)
Good question. There are a few bugs and blueprints about it, and they show as *implemented*
Yes, it's implemented and works well. I use it on Wikimedia servers when I need debug symbols for stock packages. My only gripe is that it would be nice to have awareness of the feature in higher-level tools like apt and synaptic, to allow installation of the debug symbols of any package without slowing down "apt-get update" or cluttering up the synaptic package list. I'm thinking something similar to "apt-get source".
The Debian habit of adding -dbg packages to some fraction of binary packages also clutters the package list, but with an architecture that makes apt/synaptic support unlikely.
For packages that we build ourselves, it's easier to just disable stripping.
-- Tim Starling
wikitech-l@lists.wikimedia.org