As promised, I started to dig in to the Maps documentation [1] and
started to write some diagrams. The source of those diagram is at the
end of this email, you can process that source online if you want the
pretty pictures [2] (well, not that pretty).
I should probably post those diagrams somewhere for discussion, but
not sure where it make sense...
[1]
https://wikitech.wikimedia.org/wiki/Maps
[2]
http://www.planttext.com/planttext
**Maps component diagram:**
@startuml
[Varnish]
[Kartotherian]
[Tilerator]
[TileratorUI]
[OSM]
database Cassandra
database Redis
database Postgres
OSM -> Postgres: import maps data
Tilerator -left-> Redis: consume job queue
Tilerator --> Postgres: get data to pre-generate\nvector tiles
Tilerator -> Cassandra: stores pre-generated\nvector tiles
TileratorUI ..> Redis: schedule jobs to\npre-generate tiles
Kartotherian --> Cassandra: serve tiles\nto end user
Varnish --> Kartotherian
@enduml
**Maps deployment diagram:**
@startuml
() "Maps\n(public)" as mapsP
package codfw {
package "Maps cache cluster" as cache {
node cp2003 {
[varnish-frontend] as vfe2003
[varnish-backend] as vbe2003
}
node cp2009 {
[varnish-frontend] as vfe2009
[varnish-backend] as vbe2009
}
node cp2015 {
[varnish-frontend] as vfe2015
[varnish-backend] as vbe2015
}
node cp2021 {
[varnish-frontend] as vfe2021
[varnish-backend] as vbe2021
}
}
() "Maps\n(internal)" as mapsI
node "maps-test2001\n(master)" as maps2001 {
[Kartotherian] as Kartotherian2001
[Tilerator] as Tilerator2001
[TileratorUI] as TileratorUI2001
database Cassandra as Cassandra2001
database Redis as Redis2001
database "Postgres\nmaster" as Postgres2001
Tilerator2001 -left-> Redis2001
Tilerator2001 --> Postgres2001
Tilerator2001 -> Cassandra2001
TileratorUI2001 --> Redis2001
Kartotherian2001 --> Cassandra2001
}
node "maps-test2002-4\n(slaves)" as maps2002 {
[Kartotherian] as Kartotherian20xx
[Tilerator] as Tilerator20xx
[TileratorUI] as TileratorUI20xx
database Cassandra as Cassandra20xx
database Redis as Redis20xx
database "Postgres\nslaves" as Postgres20xx
Tilerator20xx -left-> Redis20xx
Tilerator20xx --> Postgres20xx
Tilerator20xx --> Cassandra20xx
TileratorUI20xx -> Redis20xx
Kartotherian20xx --> Cassandra20xx
}
mapsI - Kartotherian2001
mapsI - Kartotherian20xx
vbe2003 -> mapsI
vbe2009 -> mapsI
vbe2015 -> mapsI
vbe2021 -> mapsI
vfe2003 --> vbe2003
vfe2009 --> vbe2009
vfe2015 --> vbe2015
vfe2021 --> vbe2021
' un comment the block below to have the mostly complete Varnish connections
' vfe2003 --> vbe2003
' vfe2003 --> vbe2009
' vfe2003 --> vbe2015
' vfe2003 --> vbe2021
'
' vfe2009 --> vbe2003
' vfe2009 --> vbe2009
' vfe2009 --> vbe2015
' vfe2009 --> vbe2021
'
' vfe2015 --> vbe2003
' vfe2015 --> vbe2009
' vfe2015 --> vbe2015
' vfe2015 --> vbe2021
'
' vfe2021 --> vbe2003
' vfe2021 --> vbe2009
' vfe2021 --> vbe2015
' vfe2021 --> vbe2021
mapsP -- vfe2003
mapsP -- vfe2009
mapsP -- vfe2015
mapsP -- vfe2021
Postgres20xx <- Postgres2001
Cassandra20xx <-> Cassandra2001
Redis20xx <-> Redis2001
note right of vfe2003
interconnections between
Varnish frontend and backend
are more complex, not showing
all this here.
end note
note right of mapsI
Need to check if this is a
LVS endpoint or if Varnish talks
directly to Mapsf
end note
}
note as n1
- unsure about what communication
there is between maps-test nodes
end note
@enduml
--
Guillaume Lederrey
Operations Engineer, Discovery
Wikimedia Foundation