Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team).
Thank you Marko and Petr for making this a reality!
Nice work, all!
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
Gabriel already said it - But i t’s fantastic to see how everyone pulled together to get this out! Thanks again to Marko and Petr for all the support in making this happen. And Jon for pushing so hard to take his idea from prototype to a RESTBase service.
I spoke with Josh about the new service earlier today and iOS is pretty excited to start playing with this to see what they can do!
I already can’t wait to see what everyone builds next quarter!
Congrats again!
On Tue, Dec 13, 2016 at 6:07 PM Gabriel Wicke gwicke@wikimedia.org wrote:
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Gabriel Wicke Principal Engineer, Wikimedia Foundation
I just mention there are a few hiccups as can be expected and it's not quite behaving as we would hope... but it's live and only going to get better :)
https://phabricator.wikimedia.org/T153122
On Tue, Dec 13, 2016 at 3:28 PM Corey Floyd cfloyd@wikimedia.org wrote:
Gabriel already said it - But i t’s fantastic to see how everyone pulled together to get this out! Thanks again to Marko and Petr for all the support in making this happen. And Jon for pushing so hard to take his idea from prototype to a RESTBase service.
I spoke with Josh about the new service earlier today and iOS is pretty excited to start playing with this to see what they can do!
I already can’t wait to see what everyone builds next quarter!
Congrats again!
On Tue, Dec 13, 2016 at 6:07 PM Gabriel Wicke gwicke@wikimedia.org wrote:
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
_______________________________________________ Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
On 14 December 2016 at 20:39, Jon Robson jrobson@wikimedia.org wrote:
I just mention there are a few hiccups as can be expected and it's not quite behaving as we would hope... but it's live and only going to get better :)
By switching the Kafka consumer to a different consuming mode the issue has been mitigated temporarily (see the ticket for a long-term solution). I feel confident that we are now good to go through the 4-week deployment freeze, technically speaking. Feel free to use this time to explore the service's results over that period and work on any product- or logic-related shortcomings (but keep in mind the no-deploy rule during the freeze).
Jon, you mentioned a script to pull results from the service every 10 minutes. Beware that its output is cached for 30 minutes in Varnish. We started with a higher caching value to ensure stability. Let's revisit this interval after the freeze, i.e. after AllHands.
Cheers, Marko
On Tue, Dec 13, 2016 at 3:28 PM Corey Floyd cfloyd@wikimedia.org wrote:
Gabriel already said it - But i t’s fantastic to see how everyone pulled together to get this out! Thanks again to Marko and Petr for all the support in making this happen. And Jon for pushing so hard to take his idea from prototype to a RESTBase service.
I spoke with Josh about the new service earlier today and iOS is pretty excited to start playing with this to see what they can do!
I already can’t wait to see what everyone builds next quarter!
Congrats again!
On Tue, Dec 13, 2016 at 6:07 PM Gabriel Wicke gwicke@wikimedia.org wrote:
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Gabriel Wicke Principal Engineer, Wikimedia Foundation
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
Very cool, good job everyone, and congratulations!
On Thu, Dec 15, 2016 at 1:45 PM, Marko Obrovac mobrovac@wikimedia.org wrote:
On 14 December 2016 at 20:39, Jon Robson jrobson@wikimedia.org wrote:
I just mention there are a few hiccups as can be expected and it's not quite behaving as we would hope... but it's live and only going to get better :)
By switching the Kafka consumer to a different consuming mode the issue has been mitigated temporarily (see the ticket for a long-term solution). I feel confident that we are now good to go through the 4-week deployment freeze, technically speaking. Feel free to use this time to explore the service's results over that period and work on any product- or logic-related shortcomings (but keep in mind the no-deploy rule during the freeze).
Jon, you mentioned a script to pull results from the service every 10 minutes. Beware that its output is cached for 30 minutes in Varnish. We started with a higher caching value to ensure stability. Let's revisit this interval after the freeze, i.e. after AllHands.
Cheers, Marko
On Tue, Dec 13, 2016 at 3:28 PM Corey Floyd cfloyd@wikimedia.org wrote:
Gabriel already said it - But i t’s fantastic to see how everyone pulled together to get this out! Thanks again to Marko and Petr for all the support in making this happen. And Jon for pushing so hard to take his idea from prototype to a RESTBase service.
I spoke with Josh about the new service earlier today and iOS is pretty excited to start playing with this to see what they can do!
I already can’t wait to see what everyone builds next quarter!
Congrats again!
On Tue, Dec 13, 2016 at 6:07 PM Gabriel Wicke gwicke@wikimedia.org wrote:
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Gabriel Wicke Principal Engineer, Wikimedia Foundation
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Marko Obrovac, PhD Senior Services Engineer Wikimedia Foundation
Yes! Great work, so happy it all came together.
On Thu, Dec 15, 2016 at 8:53 AM Joaquin Oltra Hernandez < jhernandez@wikimedia.org> wrote:
Very cool, good job everyone, and congratulations!
On Thu, Dec 15, 2016 at 1:45 PM, Marko Obrovac mobrovac@wikimedia.org wrote:
On 14 December 2016 at 20:39, Jon Robson jrobson@wikimedia.org wrote:
I just mention there are a few hiccups as can be expected and it's not quite behaving as we would hope... but it's live and only going to get better :)
https://phabricator.wikimedia.org/T153122
By switching the Kafka consumer to a different consuming mode the issue has been mitigated temporarily (see the ticket for a long-term solution). I feel confident that we are now good to go through the 4-week deployment freeze, technically speaking. Feel free to use this time to explore the service's results over that period and work on any product- or logic-related shortcomings (but keep in mind the no-deploy rule during the freeze).
Jon, you mentioned a script to pull results from the service every 10 minutes. Beware that its output is cached for 30 minutes in Varnish. We started with a higher caching value to ensure stability. Let's revisit this interval after the freeze, i.e. after AllHands.
Cheers, Marko
On Tue, Dec 13, 2016 at 3:28 PM Corey Floyd cfloyd@wikimedia.org wrote:
Gabriel already said it - But i t’s fantastic to see how everyone pulled together to get this out! Thanks again to Marko and Petr for all the support in making this happen. And Jon for pushing so hard to take his idea from prototype to a RESTBase service.
I spoke with Josh about the new service earlier today and iOS is pretty excited to start playing with this to see what they can do!
I already can’t wait to see what everyone builds next quarter!
Congrats again!
On Tue, Dec 13, 2016 at 6:07 PM Gabriel Wicke gwicke@wikimedia.org wrote:
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Gabriel Wicke Principal Engineer, Wikimedia Foundation
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Marko Obrovac, PhD Senior Services Engineer Wikimedia Foundation
--
Olga Vasileva // Product Manager // Reading Web Team https://wikimediafoundation.org/
On Thu, Dec 15, 2016 at 4:45 AM Marko Obrovac mobrovac@wikimedia.org wrote:
On 14 December 2016 at 20:39, Jon Robson jrobson@wikimedia.org wrote:
I just mention there are a few hiccups as can be expected and it's not quite behaving as we would hope... but it's live and only going to get better :)
https://phabricator.wikimedia.org/T153122
By switching the Kafka consumer to a different consuming mode the issue has been mitigated temporarily (see the ticket for a long-term solution). I feel confident that we are now good to go through the 4-week deployment freeze, technically speaking. Feel free to use this time to explore the service's results over that period and work on any product- or logic-related shortcomings (but keep in mind the no-deploy rule during the freeze).
Jon, you mentioned a script to pull results from the service every 10 minutes. Beware that its output is cached for 30 minutes in Varnish. We started with a higher caching value to ensure stability. Let's revisit this interval after the freeze, i.e. after AllHands.
Yup we should definitely revisit this. I'm yet to find overlap with the results in https://trending.wmflabs.org/ which I think is due to this delay. Right now the API tells me http://en.m.wikipedia.org/wiki/Velupillai_Prabhakaran is the most edited but the last edit to that page was 50 minutes ago. Is it possible that this lag is higher than 30 in practice?
Also, is the processing of old events (the recovery step) synchronous? I'm a little worried when we hit the API we're not processing the last 1 hour of edits before returning results.
Sadly (although maybe not so much for the world) nothing has really trending over the last few days so it's hard to verify I'm seeing what I expect us to.
Cheers, Marko
On Tue, Dec 13, 2016 at 3:28 PM Corey Floyd cfloyd@wikimedia.org wrote:
Gabriel already said it - But i t’s fantastic to see how everyone pulled together to get this out! Thanks again to Marko and Petr for all the support in making this happen. And Jon for pushing so hard to take his idea from prototype to a RESTBase service.
I spoke with Josh about the new service earlier today and iOS is pretty excited to start playing with this to see what they can do!
I already can’t wait to see what everyone builds next quarter!
Congrats again!
On Tue, Dec 13, 2016 at 6:07 PM Gabriel Wicke gwicke@wikimedia.org wrote:
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Gabriel Wicke Principal Engineer, Wikimedia Foundation
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Marko Obrovac, PhD Senior Services Engineer Wikimedia Foundation
The deploy of the fix happened around noon UTC, BTW: $ grep '^t|commit|user' .git/DEPLOY_HEAD commit: 200a7094b16ccd47373b3e7c5adcc5a121cd5a19 tag: scap/sync/2016-12-15/0001 timestamp: '2016-*12-15T12*:04:44.159321' user: mobrovac
The processing delay looks much better since the deploy: https://grafana.wikimedia.org/dashboard/db/trending-service?from=14817464410...
On Thu, Dec 15, 2016 at 10:53 AM, Jon Robson jrobson@wikimedia.org wrote:
On Thu, Dec 15, 2016 at 4:45 AM Marko Obrovac mobrovac@wikimedia.org wrote:
On 14 December 2016 at 20:39, Jon Robson jrobson@wikimedia.org wrote:
I just mention there are a few hiccups as can be expected and it's not quite behaving as we would hope... but it's live and only going to get better :)
https://phabricator.wikimedia.org/T153122
By switching the Kafka consumer to a different consuming mode the issue has been mitigated temporarily (see the ticket for a long-term solution). I feel confident that we are now good to go through the 4-week deployment freeze, technically speaking. Feel free to use this time to explore the service's results over that period and work on any product- or logic-related shortcomings (but keep in mind the no-deploy rule during the freeze).
Jon, you mentioned a script to pull results from the service every 10 minutes. Beware that its output is cached for 30 minutes in Varnish. We started with a higher caching value to ensure stability. Let's revisit this interval after the freeze, i.e. after AllHands.
Yup we should definitely revisit this. I'm yet to find overlap with the results in https://trending.wmflabs.org/ which I think is due to this delay. Right now the API tells me http://en.m.wikipedia.org/wiki/Velupillai_Prabhakaran is the most edited but the last edit to that page was 50 minutes ago. Is it possible that this lag is higher than 30 in practice?
Also, is the processing of old events (the recovery step) synchronous? I'm a little worried when we hit the API we're not processing the last 1 hour of edits before returning results.
Sadly (although maybe not so much for the world) nothing has really trending over the last few days so it's hard to verify I'm seeing what I expect us to.
Cheers, Marko
On Tue, Dec 13, 2016 at 3:28 PM Corey Floyd cfloyd@wikimedia.org wrote:
Gabriel already said it - But i t’s fantastic to see how everyone pulled together to get this out! Thanks again to Marko and Petr for all the support in making this happen. And Jon for pushing so hard to take his idea from prototype to a RESTBase service.
I spoke with Josh about the new service earlier today and iOS is pretty excited to start playing with this to see what they can do!
I already can’t wait to see what everyone builds next quarter!
Congrats again!
On Tue, Dec 13, 2016 at 6:07 PM Gabriel Wicke gwicke@wikimedia.org wrote:
This one was real quick & smooth -- it certainly landed more quickly than I expected. Petr did a great job building the specialized Kafka consumer for this service, Jon built the actual trending logic, and Marko worked with ops on the service deployment. Another great collaboration, congratulations to all involved!
I'm also looking forward to see how this data is going to be used creatively.
Cheers,
Gabriel
On Tue, Dec 13, 2016 at 2:21 PM, Adam Baso abaso@wikimedia.org wrote:
Yay! Great stuff (cc reading web team). Thank you Marko and Petr for making this a reality!
Nice work, all!
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Gabriel Wicke Principal Engineer, Wikimedia Foundation
Services mailing list Services@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/services
-- Marko Obrovac, PhD Senior Services Engineer Wikimedia Foundation