minimal hardware requirements for loading wikidata dump in Blazegraph

List overview All Threads
Download

newer

older

Significant change of Wikidata...

“Wikidata Digest, Vol 84, Issue...

Adam Sanchez

4 Jun 2019 4 Jun '19

6:16 p.m.

Hello, Does somebody know the minimal hardware requirements (disk size and RAM) for loading wikidata dump in Blazegraph? The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. The bigdata.jnl file which stores all the triples data in Blazegraph is 478G but still growing. I had 1T disk but is almost full now. Thanks, Adam

Show replies by thread

Guillaume Lederrey

4 Jun 4 Jun

6:24 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com> wrote:

...

Hello, Does somebody know the minimal hardware requirements (disk size and RAM) for loading wikidata dump in Blazegraph?

The actual hardware requirements will depend on your use case. But for comparison, our production servers are: * 16 cores (hyper threaded, 32 threads) * 128G RAM * 1.5T of SSD storage

...

The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. The bigdata.jnl file which stores all the triples data in Blazegraph is 478G but still growing. I had 1T disk but is almost full now.

The current size of our jnl file in production is ~670G. Hope that helps! Guillaume

...

Thanks, Adam _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- Guillaume Lederrey Engineering Manager, Search Platform Wikimedia Foundation UTC+2 / CEST

Marco Neumann

6:49 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

thanks Guillaume. How does that compare to the wikidata footprint of the wikidata service (SQL) not WDQS. I presume it sits in a MyISAM storage container? On Tue, Jun 4, 2019 at 11:25 AM Guillaume Lederrey <glederrey(a)wikimedia.org> wrote:

...

On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com> wrote:

Hello, Does somebody know the minimal hardware requirements (disk size and RAM) for loading wikidata dump in Blazegraph?

The actual hardware requirements will depend on your use case. But for comparison, our production servers are: * 16 cores (hyper threaded, 32 threads) * 128G RAM * 1.5T of SSD storage

The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. The bigdata.jnl file which stores all the triples data in Blazegraph is 478G but still growing. I had 1T disk but is almost full now.

The current size of our jnl file in production is ~670G. Hope that helps! Guillaume

Thanks, Adam _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- Guillaume Lederrey Engineering Manager, Search Platform Wikimedia Foundation UTC+2 / CEST _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- --- Marco Neumann KONA

Adam Sanchez

7 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

Thanks Guillaume! One question more, what is the CPU frequency (GHz)? Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit :

...

On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com> wrote:

Hello, Does somebody know the minimal hardware requirements (disk size and RAM) for loading wikidata dump in Blazegraph?

The actual hardware requirements will depend on your use case. But for comparison, our production servers are: * 16 cores (hyper threaded, 32 threads) * 128G RAM * 1.5T of SSD storage

The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. The bigdata.jnl file which stores all the triples data in Blazegraph is 478G but still growing. I had 1T disk but is almost full now.

The current size of our jnl file in production is ~670G. Hope that helps! Guillaume

Thanks, Adam _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Vi to

9:13 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

AFAIR it's a double Xeon E5-2620 v3. With modern CPUs frequency is not so significant. Vito Il giorno mar 4 giu 2019 alle ore 13:00 Adam Sanchez <a.sanchez75(a)gmail.com> ha scritto:

...

Thanks Guillaume! One question more, what is the CPU frequency (GHz)? Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit :

On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com>

wrote:

Hello, Does somebody know the minimal hardware requirements (disk size and RAM) for loading wikidata dump in Blazegraph?

The actual hardware requirements will depend on your use case. But for comparison, our production servers are: * 16 cores (hyper threaded, 32 threads) * 128G RAM * 1.5T of SSD storage

The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. The bigdata.jnl file which stores all the triples data in Blazegraph is 478G but still growing. I had 1T disk but is almost full now.

The current size of our jnl file in production is ~670G. Hope that helps! Guillaume

Thanks, Adam _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Guillaume Lederrey

10:31 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

On Tue, Jun 4, 2019 at 3:14 PM Vi to <vituzzu.wiki(a)gmail.com> wrote:

...

AFAIR it's a double Xeon E5-2620 v3. With modern CPUs frequency is not so significant.

Our latest batch of servers are: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (so v4 instead of v3, but the difference is probably minimal).

...

Vito Il giorno mar 4 giu 2019 alle ore 13:00 Adam Sanchez <a.sanchez75(a)gmail.com> ha scritto:

Thanks Guillaume! One question more, what is the CPU frequency (GHz)? Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit :

On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com> wrote:

Hello, Does somebody know the minimal hardware requirements (disk size and RAM) for loading wikidata dump in Blazegraph?

The actual hardware requirements will depend on your use case. But for comparison, our production servers are: * 16 cores (hyper threaded, 32 threads) * 128G RAM * 1.5T of SSD storage

The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. The bigdata.jnl file which stores all the triples data in Blazegraph is 478G but still growing. I had 1T disk but is almost full now.

The current size of our jnl file in production is ~670G. Hope that helps! Guillaume

Thanks, Adam _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- Guillaume Lederrey Engineering Manager, Search Platform Wikimedia Foundation UTC+2 / CEST

Vi to

10:36 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

V4 has 8 cores instead of 6. But well, it's a server grade config on purpose! Vito Il giorno mar 4 giu 2019 alle ore 16:32 Guillaume Lederrey < glederrey(a)wikimedia.org> ha scritto:

...

On Tue, Jun 4, 2019 at 3:14 PM Vi to <vituzzu.wiki(a)gmail.com> wrote:

AFAIR it's a double Xeon E5-2620 v3. With modern CPUs frequency is not so significant.

Our latest batch of servers are: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (so v4 instead of v3, but the difference is probably minimal).

Vito Il giorno mar 4 giu 2019 alle ore 13:00 Adam Sanchez <

a.sanchez75(a)gmail.com> ha scritto:

> > Thanks Guillaume! > One question more, what is the CPU frequency (GHz)? > > Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey > <glederrey(a)wikimedia.org> a écrit : > > > > On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com>

wrote:

> > Hello, > > Does somebody know the minimal hardware requirements (disk size and > RAM) for loading wikidata dump in Blazegraph? The actual hardware requirements will depend on your use case. But for comparison, our production servers are: * 16 cores (hyper threaded, 32 threads) * 128G RAM * 1.5T of SSD storage > The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. > The bigdata.jnl file which stores all the triples data in Blazegraph > is 478G but still growing. > I had 1T disk but is almost full now. The current size of our jnl file in production is ~670G. Hope that helps! Guillaume > Thanks, > > Adam > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata -- Guillaume Lederrey Engineering Manager, Search Platform Wikimedia Foundation UTC+2 / CEST _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

Adam Sanchez

20 Jun 20 Jun

8:37 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

For your information a) It took 10.2 days to load the Wikidata RDF dump (wikidata-20190513-all-BETA.ttl, 379G) in Blazegraph 2.1.5. The bigdata.jnl file turned to be 1.3T Server technical features Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz Stepping: 1 CPU MHz: 1200.476 CPU max MHz: 3000.0000 CPU min MHz: 1200.0000 BogoMIPS: 4197.65 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K RAM: 128G b) It took 43 hours to load the Wikidata RDF dump (wikidata-20190610-all-BETA.ttl, 383G) in the dev version of Virtuoso 07.20.3230. I had to patch Virtuoso because it was given the following error each time I load the RDF data 09:58:06 PL LOG: File /backup/wikidata-20190610-all-BETA.ttl error 42000 TURTLE RDF loader, line 2984680: RDFGE: RDF box with a geometry RDF type and a non-geometry content The virtuoso.db file turned to be 340G. Server technical features Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Stepping: 2 CPU MHz: 1199.920 CPU max MHz: 3800.0000 CPU min MHz: 1200.0000 BogoMIPS: 6984.39 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-11 RAM: 128G Best, Le mar. 4 juin 2019 à 16:37, Vi to <vituzzu.wiki(a)gmail.com> a écrit :

...

V4 has 8 cores instead of 6. But well, it's a server grade config on purpose! Vito Il giorno mar 4 giu 2019 alle ore 16:32 Guillaume Lederrey <glederrey(a)wikimedia.org> ha scritto:

On Tue, Jun 4, 2019 at 3:14 PM Vi to <vituzzu.wiki(a)gmail.com> wrote:

AFAIR it's a double Xeon E5-2620 v3. With modern CPUs frequency is not so significant.

Our latest batch of servers are: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (so v4 instead of v3, but the difference is probably minimal).

Vito Il giorno mar 4 giu 2019 alle ore 13:00 Adam Sanchez <a.sanchez75(a)gmail.com> ha scritto:

Thanks Guillaume! One question more, what is the CPU frequency (GHz)? Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit : > > On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com> wrote: > > > > Hello, > > > > Does somebody know the minimal hardware requirements (disk size and > > RAM) for loading wikidata dump in Blazegraph? > > The actual hardware requirements will depend on your use case. But for > comparison, our production servers are: > > * 16 cores (hyper threaded, 32 threads) > * 128G RAM > * 1.5T of SSD storage > > > The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. > > The bigdata.jnl file which stores all the triples data in Blazegraph > > is 478G but still growing. > > I had 1T disk but is almost full now. > > The current size of our jnl file in production is ~670G. > > Hope that helps! > > Guillaume > > > Thanks, > > > > Adam > > > > _______________________________________________ > > Wikidata mailing list > > Wikidata(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikidata > > > > -- > Guillaume Lederrey > Engineering Manager, Search Platform > Wikimedia Foundation > UTC+2 / CEST > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

fn＠imm.dtu.dk

21 Jun 21 Jun

12:34 a.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

Is that with SSD harddisks? Isn't the bottleneck the io traffic to the harddisks? (I suppose your are not loading into RAM?) What was you hardware configuration? best regards Finn http://people.compute.dtu.dk/faan/ On 20/06/2019 14:37, Adam Sanchez wrote:

...

V4 has 8 cores instead of 6. But well, it's a server grade config on purpose! Vito Il giorno mar 4 giu 2019 alle ore 16:32 Guillaume Lederrey <glederrey(a)wikimedia.org> ha scritto:

On Tue, Jun 4, 2019 at 3:14 PM Vi to <vituzzu.wiki(a)gmail.com> wrote:

AFAIR it's a double Xeon E5-2620 v3. With modern CPUs frequency is not so significant.

Our latest batch of servers are: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (so v4 instead of v3, but the difference is probably minimal).

Vito Il giorno mar 4 giu 2019 alle ore 13:00 Adam Sanchez <a.sanchez75(a)gmail.com> ha scritto: > > Thanks Guillaume! > One question more, what is the CPU frequency (GHz)? > > Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey > <glederrey(a)wikimedia.org> a écrit : >> >> On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com> wrote: >>> >>> Hello, >>> >>> Does somebody know the minimal hardware requirements (disk size and >>> RAM) for loading wikidata dump in Blazegraph? >> >> The actual hardware requirements will depend on your use case. But for >> comparison, our production servers are: >> >> * 16 cores (hyper threaded, 32 threads) >> * 128G RAM >> * 1.5T of SSD storage >> >>> The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. >>> The bigdata.jnl file which stores all the triples data in Blazegraph >>> is 478G but still growing. >>> I had 1T disk but is almost full now. >> >> The current size of our jnl file in production is ~670G. >> >> Hope that helps! >> >> Guillaume >> >>> Thanks, >>> >>> Adam >>> >>> _______________________________________________ >>> Wikidata mailing list >>> Wikidata(a)lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/wikidata >> >> >> >> -- >> Guillaume Lederrey >> Engineering Manager, Search Platform >> Wikimedia Foundation >> UTC+2 / CEST >> >> _______________________________________________ >> Wikidata mailing list >> Wikidata(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikidata > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

hellmann＠informatik.uni-leipzig.de

12:48 a.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

Hi Adam, the server specs you posted are not so important. What disks did you use? They should be SSD or 15k RPM SAS to make it faster. Virtuoso can parse multi thread if you split the files before loading, but hdd speed is still the bottleneck. Sebastian On June 20, 2019 2:37:16 PM GMT+02:00, Adam Sanchez <a.sanchez75(a)gmail.com> wrote:

...

V4 has 8 cores instead of 6. But well, it's a server grade config on purpose! Vito Il giorno mar 4 giu 2019 alle ore 16:32 Guillaume Lederrey

<glederrey(a)wikimedia.org> ha scritto:

> > On Tue, Jun 4, 2019 at 3:14 PM Vi to <vituzzu.wiki(a)gmail.com> wrote: > > > > AFAIR it's a double Xeon E5-2620 v3. > > With modern CPUs frequency is not so significant. > > Our latest batch of servers are: Intel(R) Xeon(R) CPU E5-2620 v4 @ > 2.10GHz (so v4 instead of v3, but the difference is probably

minimal).

> > > Vito > > > > Il giorno mar 4 giu 2019 alle ore 13:00 Adam Sanchez

<a.sanchez75(a)gmail.com> ha scritto:

> >> > >> Thanks Guillaume! > >> One question more, what is the CPU frequency (GHz)? > >> > >> Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey > >> <glederrey(a)wikimedia.org> a écrit : > >> > > >> > On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez

<a.sanchez75(a)gmail.com> wrote:

> >> > > > >> > > Hello, > >> > > > >> > > Does somebody know the minimal hardware requirements (disk

size and

> >> > > RAM) for loading wikidata dump in Blazegraph? > >> > > >> > The actual hardware requirements will depend on your use case.

But for

> >> > comparison, our production servers are: > >> > > >> > * 16 cores (hyper threaded, 32 threads) > >> > * 128G RAM > >> > * 1.5T of SSD storage > >> > > >> > > The downloaded dump file wikidata-20190513-all-BETA.ttl is

379G.

> >> > > The bigdata.jnl file which stores all the triples data in

Blazegraph

> > > is 478G but still growing. > > > I had 1T disk but is almost full now. > > > > The current size of our jnl file in production is ~670G. > > > > Hope that helps! > > > > Guillaume > > > > > Thanks, > > > > > > Adam > > > > > > _______________________________________________ > > > Wikidata mailing list > > > Wikidata(a)lists.wikimedia.org > > > https://lists.wikimedia.org/mailman/listinfo/wikidata > > > > > > > > -- > > Guillaume Lederrey > > Engineering Manager, Search Platform > > Wikimedia Foundation > > UTC+2 / CEST > > > > _______________________________________________ > > Wikidata mailing list > > Wikidata(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikidata > > _______________________________________________ > Wikidata mailing list > Wikidata(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

_______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

Kingsley Idehen

1:55 a.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

On 6/20/19 12:48 PM, hellmann(a)informatik.uni-leipzig.de wrote:

...

Yep! And if the shared-nothing cluster edition is in use, you can run the bulk loaders in parallel across each of the nodes in the clusters which will reduce the load time too. We've cluster configurations behind our LOD instance where all of DBpedia was loaded in 15 minutes flat, and I don't mean via some massive cluster setup just what we have behind our LOD Cloud cache instance :) Kingsley

...

On June 20, 2019 2:37:16 PM GMT+02:00, Adam Sanchez <a.sanchez75(a)gmail.com> wrote: For your information a) It took 10.2 days to load the Wikidata RDF dump (wikidata-20190513-all-BETA.ttl, 379G) in Blazegraph 2.1.5. The bigdata.jnl file turned to be 1.3T Server technical features Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz Stepping: 1 CPU MHz: 1200.476 CPU max MHz: 3000.0000 CPU min MHz: 1200.0000 BogoMIPS: 4197.65 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K RAM: 128G b) It took 43 hours to load the Wikidata RDF dump (wikidata-20190610-all-BETA.ttl, 383G) in the dev version of Virtuoso 07.20.3230. I had to patch Virtuoso because it was given the following error each time I load the RDF data 09:58:06 PL LOG: File /backup/wikidata-20190610-all-BETA.ttl error 42000 TURTLE RDF loader, line 2984680: RDFGE: RDF box with a geometry RDF type and a non-geometry content The virtuoso.db file turned to be 340G. Server technical features Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Stepping: 2 CPU MHz: 1199.920 CPU max MHz: 3800.0000 CPU min MHz: 1200.0000 BogoMIPS: 6984.39 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-11 RAM: 128G Best, Le mar. 4 juin 2019 à 16:37, Vi to <vituzzu.wiki(a)gmail.com> a écrit : V4 has 8 cores instead of 6. But well, it's a server grade config on purpose! Vito Il giorno mar 4 giu 2019 alle ore 16:32 Guillaume Lederrey <glederrey(a)wikimedia.org> ha scritto: On Tue, Jun 4, 2019 at 3:14 PM Vi to <vituzzu.wiki(a)gmail.com> wrote: AFAIR it's a double Xeon E5-2620 v3. With modern CPUs frequency is not so significant. Our latest batch of servers are: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (so v4 instead of v3, but the difference is probably minimal). Vito Il giorno mar 4 giu 2019 alle ore 13:00 Adam Sanchez <a.sanchez75(a)gmail.com> ha scritto: Thanks Guillaume! One question more, what is the CPU frequency (GHz)? Le mar. 4 juin 2019 à 12:25, Guillaume Lederrey <glederrey(a)wikimedia.org> a écrit : On Tue, Jun 4, 2019 at 12:18 PM Adam Sanchez <a.sanchez75(a)gmail.com> wrote: Hello, Does somebody know the minimal hardware requirements (disk size and RAM) for loading wikidata dump in Blazegraph? The actual hardware requirements will depend on your use case. But for comparison, our production servers are: * 16 cores (hyper threaded, 32 threads) * 128G RAM * 1.5T of SSD storage The downloaded dump file wikidata-20190513-all-BETA.ttl is 379G. The bigdata.jnl file which stores all the triples data in Blazegraph is 478G but still growing. I had 1T disk but is almost full now. The current size of our jnl file in production is ~670G. Hope that helps! Guillaume Thanks, Adam ------------------------------------------------------------------------ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata -- Guillaume Lederrey Engineering Manager, Search Platform Wikimedia Foundation UTC+2 / CEST ------------------------------------------------------------------------ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata ------------------------------------------------------------------------ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata ------------------------------------------------------------------------ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata -- Guillaume Lederrey Engineering Manager, Search Platform Wikimedia Foundation UTC+2 / CEST ------------------------------------------------------------------------ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata ------------------------------------------------------------------------ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata ------------------------------------------------------------------------ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata

-- Regards, Kingsley Idehen Founder & CEO OpenLink Software Home Page: http://www.openlinksw.com Community Support: https://community.openlinksw.com Weblogs (Blogs): Company Blog: https://medium.com/openlink-software-blog Virtuoso Blog: https://medium.com/virtuoso-blog Data Access Drivers Blog: https://medium.com/openlink-odbc-jdbc-ado-net-data-access-drivers Personal Weblogs (Blogs): Medium Blog: https://medium.com/@kidehen Legacy Blogs: http://www.openlinksw.com/blog/~kidehen/ http://kidehen.blogspot.com Profile Pages: Pinterest: https://www.pinterest.com/kidehen/ Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen Twitter: https://twitter.com/kidehen Google+: https://plus.google.com/+KingsleyIdehen/about LinkedIn: http://www.linkedin.com/in/kidehen Web Identities (WebID): Personal: http://kingsley.idehen.net/public_home/kidehen/profile.ttl#i : http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this

Ted Thibodeau Jr

25 Jun 25 Jun

10 p.m.

New subject: minimal hardware requirements for loading wikidata dump in Blazegraph

On Jun 20, 2019, at 08:37 AM, Adam Sanchez <a.sanchez75(a)gmail.com> wrote:

...

For your information ... b) It took 43 hours to load the Wikidata RDF dump (wikidata-20190610-all-BETA.ttl, 383G) in the dev version of Virtuoso 07.20.3230. I had to patch Virtuoso because it was given the following error each time I load the RDF data 09:58:06 PL LOG: File /backup/wikidata-20190610-all-BETA.ttl error 42000 TURTLE RDF loader, line 2984680: RDFGE: RDF box with a geometry RDF type and a non-geometry content The virtuoso.db file turned to be 340G. Server technical features Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz Stepping: 2 CPU MHz: 1199.920 CPU max MHz: 3800.0000 CPU min MHz: 1200.0000 BogoMIPS: 6984.39 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 15360K NUMA node0 CPU(s): 0-11 RAM: 128G Best,

Hi, Adam -- We're quite interested in the time your Wikidata load took on Virtuoso, as it seems rather slow, given our experience with other large (and much larger!) data sets. The hardware information you provided focused primarily on the processors -- but RAM and disk details are much more important to data loads. Also, there are some significant Virtuoso configuration settings (in the INI file) which have an impact. We'd like to get the info that would let us fill in the blanks on this spreadsheet (itself a work in progress), so we can do some analysis, and likely provide some tuning hints that would bring the Virtuoso Wikidata load time down significantly. https://docs.google.com/spreadsheets/d/1-stlTC_WJmMU3xA_NxA1tSLHw6_sbpjff-5… You can see the settings in use for some other deployments, on the "Current" tab, which may in themselves show you some places you could improve things immediately. Last, we would appreciate knowing exactly what you patched to get around the geodata error, as there are a few open issues along those lines, which are also works in progress. Thanks, Ted -- A: Yes. http://www.idallen.com/topposting.html | Q: Are you sure? | | A: Because it reverses the logical flow of conversation. | | | Q: Why is top posting frowned upon? Ted Thibodeau, Jr. // voice +1-781-273-0900 x32 Senior Support & Evangelism // mailto:tthibodeau@openlinksw.com // http://twitter.com/TallTed OpenLink Software, Inc. // http://www.openlinksw.com/ 20 Burlington Mall Road, Suite 322, Burlington MA 01803 Weblog -- http://www.openlinksw.com/blogs/ Community -- https://community.openlinksw.com/ LinkedIn -- http://www.linkedin.com/company/openlink-software/ Twitter -- http://twitter.com/OpenLink Facebook -- http://www.facebook.com/OpenLinkSoftware Universal Data Access, Integration, and Management Technology Providers

1766

days inactive

1787

days old

wikidata@lists.wikimedia.org

Manage subscription

11 comments

8 participants

tags (0)

participants (8)

Adam Sanchez
fn＠imm.dtu.dk
Guillaume Lederrey
hellmann＠informatik.uni-leipzig.de
Kingsley Idehen
Marco Neumann
Ted Thibodeau Jr
Vi to