pyvoa.jsondb_parser

Parsing of the JSON database descriptions.

MetaInfo reads the description files shipped under pyvoa/data/, one per database, and answers what is available and what each one declares. DataParser downloads the datasets a description names, normalises their columns to pyvoa’s own, resolves their locations against the matching geography and concatenates the result into a single long DataFrame.

Adding a database means adding a JSON file here, not writing python.

Project : pyvoa Authors : Tristan Beau, Julien Browaeys, Olivier Dadoun Copyright ©pyvoa_org License : see the joint LICENSE file https://pyvoa.org/

class pyvoa.jsondb_parser.MetaInfo

Bases: object

The catalogue of the databases pyvoa ships.

Reads the JSON description files under pyvoa/data/, one per database, and answers what is available and what a given database declares: its geography, its datasets and their urls, and the columns each of them carries. Adding a database means adding a file here, not writing code.

__init__()

Describe the epidemiological variables of a database.

For the database called namedb, returns a dictionary keyed on the epidemiological variable, whose values are the new variable name for pyvoa’s purposes (an empty string by default), a description of the variable (an empty string by default, though describing it is strongly recommended), the url of the csv the variable is read from, and the url of the master page where a general description may be found (an empty string by default).

static parsejson(file)

Parse the description of the selected database, in json format.

The json description reads like this:

"header": "Some Header",
"geoinfo": {
            "granularity": "country / region / subregion ",
            "iso3": "World / Europe /country name / ..."
            },
"columns / rows ": [
        {
                "name":"XXX",
                "alias":"XXX",
                "description":"Some description of the variable",
                "url": "https://XXX",
                "urlmaster":"https://XXX"
        },
        ...
        ]
}

where columns / rows are the ones to keep from the database, ‘name’ is the name of the variable in pyvoa, ‘alias’ its original name in the database, ‘url’ the location of the database and ‘urlmaster’ an optional master url.

static getallmetadata()

List every valid json file of the json folder.

Returns:

The database name as the key, and the parsed json file as the value, or the error if the file does not exist or is not valid. Only valid json files are returned.

Return type:

dict

getcurrentmetadata(namedb)

Return current meta information from the json file i.e from “namedb”.json.

getcurrentmetadatawhich(dico)

Retrieve the “which” values from the parsed json.

They are the ones defined by the ‘name’ keyword in the json file, plus, for a wide file, its ‘namedata’ – or the ‘renamedata’ that replaces it.

static checkmetadatastructure(metastructure)

Check that the mandatory json metadata keys are present.

Some metadata are mandatory in the json file; this checks that all of them are there.

Returns:

A 2D list: sig (1 ok, 0 not good) and the message.

Return type:

list

class pyvoa.jsondb_parser.DataParser(namedb)

Bases: object

One database, parsed from its JSON description into a DataFrame.

Downloads the datasets the description declares, normalises their columns to pyvoa’s names, resolves their locations against the matching geography, and concatenates the result into a single long DataFrame indexed by date and location. The keyword definitions and source urls are kept alongside, so that a variable can be traced back to the file and the provider it came from.

__init__(namedb)

Load one database and parse it.

Reads the JSON description of namedb, picks the geography that matches its declared granularity – a GeoManager for a world-wide database, a GeoCountry for a national one – and parses the data straight away, so a DataParser is usable as soon as it is built.

Parameters:

namedb (str) – the database name, as listed by MetaInfo.

Raises:

PyvoaError – if the granularity is not one of country, region or subregion, or if parsing fails, which usually means the upstream file changed shape.

get_parsing()

Parse the json file loaded by the init function (self.metadata).

Returns a pandas with the columns ‘date’, ‘where’, ‘code’, ‘geometry’, ‘from_db’ and one column per variable selected in the json file. “where” and “code” go through the geo methods, to assure a good standardization. Every location of the geography is present on every date, with missing values where the source says nothing; ‘from_db’, a bool column, tells the locations the source reports from those it does not. A location the geometry file lacks carries an empty geometry rather than none.

get_db()

Return the current covid19 database selected. See get_available_database() for full list.

get_geo()

Return the geography object this database is resolved against.

Return type:

A GeoManager for a world-wide database, a GeoCountry otherwise.

get_world_boolean()

Tell whether this database is world-wide rather than national.

Returns:

True when its granularity is ‘country’.

Return type:

bool

get_locations()

Return the locations available in the current database.

Countries, regions or subregions, standardized through the geo methods.

get_dates()

Return all dates available in the current database as datetime format.

get_available_keywords()

Return the variables of the database, the default ‘which’ first.

See tools.prioritize_keyword for which one comes first.

get_url()

Return all the url which have been parsed for the database selected.

get_keyword_definition(which)

Return available keywords (originally named original keywords) definition.

get_keyword_url(which)

Return the url the given variable was parsed from.

Parameters:

which (str) – the variable name.

Return type:

The url of the file this variable comes from.

Raises:

PyvoaError – if which is missing or is not a variable of this database.

get_dbdescription()

Return available information concerning the db selected.

get_maingeopandas()

Return the parsing of the data + the geometry description as a geopandas.