Hello Thiemo. I updated the document. Look You the document or the document changes. I think, for the low number values is better storing as text. Example, one reason, the RAW data have lower memory size. Example for input "1 15 85" is the test size 7 B, but in memory format would be minimal 3 input values * (minimal) 4 bytes per one value = 12 bytes. The binary dump data would be compress (for the zero bytes in RAW data) both as a text data. The text format is more human format as like to use, Example for the programmer or to use in the spreadsheet calculator. Dušan Kreheľ 2022-09-03 11:16 GMT+02:00, Thiemo Kreuz <thiemo.kreuz(a)wikimedia.de>de>: > Hello Dušan, > > I find this really fascinating. Unfortunately, it looks like the > article doesn't explain the proposed format. Where is the domain in > the new format? What does "DAY_HOUR" mean? What's the difference > between "DAY_HOUR2", "DAY2_HOUR", and "DAY2_HOUR2"? What is the file > naming scheme for the new format? > > Being fascinated by file formats myself I also wonder. Why not make it > binary? > > Kind regards > Thiemo > _______________________________________________ > Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org > To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Our pageview dumps were in the middle of a refactor when our team changed a lot. We haven't been able to finish it, but we do actually have a well-compressed version that we just haven't properly launched as a new dataset. I'm working on prioritizing that. On Sun, Sep 4, 2022 at 02:58 Gergő Tisza <gtisza(a)gmail.com> wrote:

Hi Dušan, I added the details on pageviews_complete to the talk page on your proposal <https://en.wikipedia.org/w/index.php?title=User_talk:Du%C5%A1an_Krehe%C4%BE/Signpost_draft:New_pageview_dump_export_format_(concept)&oldid=1108690384>. Please let me know if it's still confusing.

The big update of the article is done. Please, You look. Gergő Tisza: The current fresh hour format can remain. Later it can be converted to another format. And thus be more suitable for others. 2022-09-18 22:35 GMT+02:00, Dušan Kreheľ &lt;dusankrehel(a)gmail.com&gt;om>: > I have updated the document. I added the export of human pageviews for > year 2021. The statistics are in the article. A download link has been > added. > > Dan Andreescu: None problem was to understand You. > > 2022-09-05 21:48 GMT+02:00, Dan Andreescu &lt;dandreescu(a)wikimedia.org&gt;rg>: >> Hi Dušan, >> >> I added the details on pageviews_complete to the talk page on your >> proposal >> <https://en.wikipedia.org/w/index.php?title=User_talk:Du%C5%A1an_Krehe%C4%BE/Signpost_draft:New_pageview_dump_export_format_(concept)&oldid=1108690384>. >> Please let me know if it's still confusing. >> >

A link to the source code has been added. @Dan Andreescu: The format is correct now. The annual summary is a typical basic statistical interval, and we save time by merging. The file size problem disappears if the file is split by wík. And the skwiki has only 49MB for the year 2021, which does not require the level of the end user who processes them for their purpose. 2022-10-06 19:31 GMT+02:00, Dan Andreescu &lt;dandreescu(a)wikimedia.org&gt;rg>: > @Dušan Kreheľ: I think there's a misunderstanding. I read your re-written > article. In it, you say that the current format is: > > domain_code page_title count_views total_response_size > > For an example, you give this: > > sk Kreheľ 2 0 > > But, actually, that format is deprecated and the new format is pageviews > complete, which looks like this: > > sk.wikipedia Kreheľ null desktop 13 B2D2G2J2O2T1V1X1 > > The B2D2G2J2O2T1V1X1 is exactly the kind of encoding you're talking about, > and no 0-values are present. > > You made the point that we are missing a yearly rollup in this new format. > This would be quite a large file, but if there's a good use case for such a > dump, a request in phabricator is a good way to proceed. > > On Sat, Oct 1, 2022 at 9:58 AM Dušan Kreheľ &lt;dusankrehel(a)gmail.com&gt; wrote: > >> The big update of the article is done. Please, You look. >> >> Gergő Tisza: The current fresh hour format can remain. Later it can be >> converted to another format. And thus be more suitable for others. >> >> 2022-09-18 22:35 GMT+02:00, Dušan Kreheľ &lt;dusankrehel(a)gmail.com&gt;om>: >> > I have updated the document. I added the export of human pageviews for >> > year 2021. The statistics are in the article. A download link has been >> > added. >> > >> > Dan Andreescu: None problem was to understand You. >> > >> > 2022-09-05 21:48 GMT+02:00, Dan Andreescu &lt;dandreescu(a)wikimedia.org&gt;rg>: >> >> Hi Dušan, >> >> >> >> I added the details on pageviews_complete to the talk page on your >> >> proposal >> >> < >> https://en.wikipedia.org/w/index.php?title=User_talk:Du%C5%A1an_Krehe%C4%BE… >> >. >> >> Please let me know if it's still confusing. >> >> >> > >> _______________________________________________ >> Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org >> To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org >> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >

The article is for me done now. D. K. 2022-11-08 21:32 GMT+01:00, Dušan Kreheľ &lt;dusankrehel(a)gmail.com&gt;om>: > [Fix]: > > A link to the source code has been added. > > @Dan Andreescu: The format is correct. The annual summary is a typical > basic statistical interval, and we save time by merging. The file size > problem disappears if the file is split by local wikis. And the skwiki > is only 49MB for the year 2021, which does not require a more > demanding level of the end user who processes them for their purpose. > > 2022-11-08 21:30 GMT+01:00, Dušan Kreheľ &lt;dusankrehel(a)gmail.com&gt;om>: >> A link to the source code has been added. >> >> @Dan Andreescu: The format is correct now. The annual summary is a >> typical basic statistical interval, and we save time by merging. The >> file size problem disappears if the file is split by wík. And the >> skwiki has only 49MB for the year 2021, which does not require the >> level of the end user who processes them for their purpose. >> >> 2022-10-06 19:31 GMT+02:00, Dan Andreescu &lt;dandreescu(a)wikimedia.org&gt;rg>: >>> @Dušan Kreheľ: I think there's a misunderstanding. I read your >>> re-written >>> article. In it, you say that the current format is: >>> >>> domain_code page_title count_views total_response_size >>> >>> For an example, you give this: >>> >>> sk Kreheľ 2 0 >>> >>> But, actually, that format is deprecated and the new format is pageviews >>> complete, which looks like this: >>> >>> sk.wikipedia Kreheľ null desktop 13 B2D2G2J2O2T1V1X1 >>> >>> The B2D2G2J2O2T1V1X1 is exactly the kind of encoding you're talking >>> about, >>> and no 0-values are present. >>> >>> You made the point that we are missing a yearly rollup in this new >>> format. >>> This would be quite a large file, but if there's a good use case for >>> such >>> a >>> dump, a request in phabricator is a good way to proceed. >>> >>> On Sat, Oct 1, 2022 at 9:58 AM Dušan Kreheľ &lt;dusankrehel(a)gmail.com&gt; >>> wrote: >>> >>>> The big update of the article is done. Please, You look. >>>> >>>> Gergő Tisza: The current fresh hour format can remain. Later it can be >>>> converted to another format. And thus be more suitable for others. >>>> >>>> 2022-09-18 22:35 GMT+02:00, Dušan Kreheľ &lt;dusankrehel(a)gmail.com&gt;om>: >>>> > I have updated the document. I added the export of human pageviews >>>> > for >>>> > year 2021. The statistics are in the article. A download link has >>>> > been >>>> > added. >>>> > >>>> > Dan Andreescu: None problem was to understand You. >>>> > >>>> > 2022-09-05 21:48 GMT+02:00, Dan Andreescu &lt;dandreescu(a)wikimedia.org&gt;rg>: >>>> >> Hi Dušan, >>>> >> >>>> >> I added the details on pageviews_complete to the talk page on your >>>> >> proposal >>>> >> < >>>> https://en.wikipedia.org/w/index.php?title=User_talk:Du%C5%A1an_Krehe%C4%BE… >>>> >. >>>> >> Please let me know if it's still confusing. >>>> >> >>>> > >>>> _______________________________________________ >>>> Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org >>>> To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org >>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ >>> >> >