pyvoa.tools
Utility functions shared across pyvoa.
A swiss knife of helpers, covering verbose and warning mode management, kwargs analysis, filling the nan values of a given pandas, date parsing and validation, and the automatic file caching system.
The _verbose_mode variable should be set to 0 if no printing output is
needed. The default value is 1, printing information to stdout; the value 2
grants debug level information printing.
Project : pyvoa Authors : Tristan Beau, Julien Browaeys, Olivier Dadoun Copyright ©pyvoa_org License : see the joint LICENSE file https://pyvoa.org/
- pyvoa.tools.get_live_mode()[source]
Say whether the live upstream sources are in use.
- Returns:
True when the live upstream sources are used, False when the frozen Zenodo archive is.
- Return type:
- pyvoa.tools.set_live_mode(live=True)[source]
Set the data source mode.
If live is True, downloads are made from the original upstream sources, which give the most recent data available but may have moved, changed format or disappeared. If live is False (default), everything is read from the frozen Zenodo archive, which guarantees reproducible results.
- pyvoa.tools.info(*args)[source]
Print to stdout, if the verbosity is 1 or more.
Takes the same arguments as the builtin print function.
- pyvoa.tools.verb(*args)[source]
Print to stdout, if the verbosity is 2 (debug).
Takes the same arguments as the builtin print function.
- pyvoa.tools.kwargs_keystesting(given_args, expected_args, error_string)[source]
Test that the given kwargs keys are the expected ones.
- Raises:
PyvoaError – With error_string, if a key is not expected.
- pyvoa.tools.debug(value, message='')[source]
Print a value between two rules, for debugging at the console.
Unconditional, unlike verb() and info(): it ignores the verbosity, so it is meant for temporary use while tracking a problem down, not for output the library ships.
- Parameters:
value – whatever is to be shown.
message (
str) – a label printed next to it.
- pyvoa.tools.kwargs_test(given_args, expected_args, error_string)[source]
Test that the given kwargs keys are the expected ones.
- Raises:
PyvoaError – With error_string, if a key is not expected.
- pyvoa.tools.kwargs_values_testing(given_values, expected_values, error_string)[source]
Check if values are in allowed list.
- pyvoa.tools.kwargs_keyvaluestesting(given_kargs, expected_kargs, hiddenkeys, error_string)[source]
Test that the given kwargs keys and their values are the expected ones.
- Raises:
PyvoaError – With error_string, if a key or one of its values is not expected.
- pyvoa.tools.tostdstring(s)[source]
Standardization of string for country,region or subregion tests.
- pyvoa.tools.shorten_locations(where, maxletters)[source]
Cut long location names down to a chart label.
A name longer than
maxletterskeeps its firstmaxletterscharacters, followed by ‘…’. Two names cut to the same label keep their full length instead, since a chart groups its series by label and would otherwise draw two locations as one.- Parameters:
where (
pandas.Series) – The location names.maxletters (
int) – The length past which a name is cut; below 5, 5 is used.
- Returns:
The labels, aligned on
where.- Return type:
- pyvoa.tools.scientific_label(value, digits=3)[source]
Write a value as a map label.
The value keeps
digitssignificant figures. Once rounded, a value between 0.001 and 999 in magnitude is written plainly, without its trailing zeros (570,4.5,0.0123); any other is written in scientific notation, the power of ten in superscript,5.49×10⁷, the way the colour bars of the maps write it.
- pyvoa.tools.fill_missing_dates(p, date_field='date', loc_field='where', d1=None, d2=None)[source]
Fill the input pandas dataframe p with missing dates.
Every location is given every day between d1 and d2. A day the source says nothing about is left missing, never carried over from the day before: a column of increments is cumulated afterwards, and forward filling the running totals is
GPDBuilder.get_stats’s business. A timezone on the dates is dropped first, the dates being calendar days.- Parameters:
p (
pandas.DataFrame) – The table, one row per date and location.date_field (
str) – The names of its date and location columns, ‘date’ and ‘where’ by default.loc_field (
str) – The names of its date and location columns, ‘date’ and ‘where’ by default.d1 (
datetime.date, optional) – The first and last days. Default to the first and last dates of p.d2 (
datetime.date, optional) – The first and last days. Default to the first and last dates of p.
- Returns:
The table on the full daily range, a row for every location and day.
- Return type:
- Raises:
PyvoaError – If p is not a DataFrame, lacks either column, or d1 is after d2.
- pyvoa.tools.check_valid_date(date)[source]
Check that a string is a valid date.
The format is day/month/year, with 2 digits for the day, 2 for the month and 4 for the year.
- pyvoa.tools.extract_dates(when)[source]
Extract the date range from a when argument.
Expects None, or 1 or 2 dates separated by ‘:’, given as a string. Two dates must be ordered. A single date is taken as the latest one. With None, the oldest date is 01/01/0001 and the newest is today.
- Returns:
The oldest and the newest date.
- Return type:
- pyvoa.tools.week_to_date(whenstr)[source]
Convert a week to a date.
Two cases: a rolling week, in the Y-M-D-Y-M-D format, returns the middle date; a one-week datum, Wnumber, returns the monday of that week number.
- pyvoa.tools.exists_from_url(path)[source]
Check that the url of a file responds.
- Returns:
True if the url answers with an ok status.
- Return type:
- pyvoa.tools.get_local_from_url(url, expiration_time=0, suffix='', live=None)[source]
“Download data from the given url and store it into a local file.
If the expiration time is 0 (default), the data will never be downloaded anymore if available. If the expiration time is < 0, it forces to download the file. If the expiration time (in seconds) is lower than time difference between now and last modification time of the file, the file is downloaded.
One may add a suffix to the local filename if known.
live overrides the global live mode for this one download: True for a file pyvoa has no archived copy of, which is always fetched from its provider. None, the default, follows get_live_mode().
- pyvoa.tools.all_or_none_lists(my_list)[source]
Tell whether the elements of my_list are consistently typed.
True when none of them is a list, or when all of them are; False when lists and non-lists are mixed. front uses it on ‘where’ to refuse a selection that mixes single locations and clusters of locations.
- pyvoa.tools.getnonnegfunc(mypd, which)[source]
From a mypd pandas and a which value return non negative values.
- pyvoa.tools.return_nonan_dates_pandas(df=None, field=None)[source]
Check if for last date all values are nan, if yes check previous date and loop until false.
- pyvoa.tools.readpkl(filepkl)[source]
Load a pickled object from the pyvoa cache directory.
- Parameters:
filepkl – filename to read (without path)
- Return type:
Unpickled object- Raises:
PyvoaError – if file doesn’t exist or can’t be loaded
- pyvoa.tools.dumppkl(filepkl, whattodump)[source]
Pickle an object into the package cache directory.
- Parameters:
filepkl (
str) – the file name, created under the cache path.whattodump – the object to pickle.
- Raises:
PyvoaError – if either argument is None.
- pyvoa.tools.prioritize_keyword(listwhich)[source]
Put the default ‘which’ at the head of a list of variables.
The default is the death count when there is one: the first of ‘tot_deaths’, ‘total_deaths’, ‘tot_dc’, ‘total_dc’ and ‘tot_dchosp’ (the hospital deaths of spf) found in the list. Failing that, it is the first cumulative variable, named
tot_...ortotal_..., in the order of the list; and failing that, the first entry.
- pyvoa.tools.wrap_antimeridian(geom, lon_0=0)[source]
Bring a geometry back within 180 degrees of a central meridian.
GeoInfo pushes Russia, Fiji, New Zealand and Samoa east of 180 degrees, and the USA west of -180, so that a Mercator map does not cut them in two. A projection that is not cylindrical wraps a polygon that crosses its own antimeridian round the world, drawing a band across the whole map: the geometry is cut there instead, and each piece moved back by 360 degrees.
- Parameters:
- Returns:
The same area, within [lon_0 - 180, lon_0 + 180]; an empty or missing geometry, and one already within, is returned as it is.
- Return type:
shapely geometryorNone
- pyvoa.tools.equal_area_projection(geometry, in_degrees)[source]
Project geometries onto Eckert IV, an equal-area projection.
The geometries are brought to degrees and projected onto Eckert IV, centred on 0 for a map spanning more than half the world, on the middle of the data otherwise, so that a national map is not sheared. That middle is taken before any cut, on the geometries as GeoInfo shifts them, which keeps a country across 180 degrees contiguous: the USA, Alaska included, centre near -127. They are then cut at the antimeridian of that projection (see
wrap_antimeridian()).- Parameters:
geometry (
geopandas.GeoSeries) – The geometries, whatever crs they are labelled with.in_degrees (
bool) – True if they are in degrees, False if in Web Mercator metres, asfront.get()hands them out.
- Returns:
The geometries in Eckert IV metres, with that crs.
- Return type:
- pyvoa.tools.projection_half_extent(crs)[source]
Return how far a pseudo-cylindrical projection reaches from its centre.
- Parameters:
crs (
pyproj.CRSorstr) – An Eckert IV crs, asequal_area_projection()gives; read through geopandas, pyproj not being a dependency of its own.- Returns:
(half width, half height): where 180 degrees from the central meridian on the equator, and the pole, fall.
- Return type:
- pyvoa.tools.convertmercator(gdf)[source]
Work around the bokeh antimeridian tile problem.
Trick found by dadoun, see https://discourse.bokeh.org/t/bokeh-tile-antimeridian-problem/6978.
- pyvoa.tools.get_polycoords(geopandasrow)[source]
Return the exterior coordinates of a geopandas row.
Takes a row of a geopandas, as given by
for index, row in geopdwd.iterrows().
- pyvoa.tools.min_max_range(a_min, a_max)[source]
Return a cleverly rounded min and max of the data.
Useful for a histogram range and for a colormap.
- pyvoa.tools.blinking_centered_text(typemsg, message, blinking=False, text_color='white', bg_color='red')[source]
Display a banner message, centred and coloured.
Detects the environment and renders accordingly: HTML in a Jupyter notebook or in Colab, ANSI escape codes on a plain terminal. This is what gives PyvoaError, PyvoaWarning and PyvoaInfo their coloured banner.
- Parameters:
typemsg (
str) – the heading, such as ‘PYVOA Error !’.message (
str) – the text below it.blinking (
bool) – whether the banner blinks. Errors do, the rest do not.text_color (
str) – a name from the internal colour map, or a value the display understands.bg_color (
str) – likewise, for the background.
- pyvoa.tools.PyvoaInfo(message)[source]
Print an informational banner, if the verbosity is 2 (debug).
Not an exception, despite the name: it prints and returns. See PyvoaError for the one type the library raises.
- Parameters:
message (
str) – the text to show.
- pyvoa.tools.PyvoaWarning(message)[source]
Print a warning banner, if the verbosity is 1 or more.
Not an exception, despite the name: it prints and returns. See PyvoaError for the one type the library raises.
- Parameters:
message (
str) – the text to show.
- exception pyvoa.tools.PyvoaError(*args)[source]
Bases:
ExceptionThe single exception type raised by pyvoa.
Displaying the message is done at construction time, so that the usual coloured banner is shown to the end user even when the traceback itself is hidden (notebooks, front end helpers). Being a genuine exception, it can also be caught and inspected, which the previous implementation (a function calling sys.exit) did not allow.