Apply filters to series (time series editor)
This is a meta-issue. Links to sub-issues will be added in the text.
Projects:
- https://git.nomics.world/dbnomics-editor/editor-api
- https://git.nomics.world/dbnomics-editor/editor-ui
Presentation
- the "time series editor" is a project related to DBnomics aiming to transform time series by applying to them a composition of filters
- this project is not necessarily implemented via enhancements to DBnomics, but via new source code repositories
- the application source code MUST be generic enough to allow defining new filters later
- computation results are never stored
- analytics SHOULD be setup with analytics.nomics.world
- it COULD be usable with any data like a CSV file, not especially tied to DBnomics, but it would require more work (i.e. create an UI to select the "period" and "value" columns)
Filters
- Define filters and their parameters in a YAML file
Example filters.yml
:
filters:
- name: aggregate
parameters:
- name: frequency
type: string
- name: method
type: string
options:
- average
- sum
- end_of_period
- name: interpolate
parameters:
- name: method
type: string
options:
- linear
- spline
- name: x13
Agrégation temporelle
- Menu permettant de choisir une fréquence plus basse que la fréquence d'origine
- Menu permettant de choisir la méthode d'agrégation (average, sum, end of period)
- Exemple : Edit graph sur Fred, https://fred.stlouisfed.org/series/A191RL1Q225SBEA
Interpolation
- Linéaire et spline.
- Exemples R pour les fonctions d'interpolation linéaire (https://www.rdocumentation.org/packages/stats/versions/3.5.3/topics/approxfun) et spline (https://www.rdocumentation.org/packages/stats/versions/3.5.3/topics/splinefun)
Désaisonalisation au moyen de X13
Il convient de trouver les arguments les plus utiles parmi une multitude d'options.
- Documentation générale : https://www.census.gov/srd/www/x13as/
- Exemple de site shiny http://www.seasonal.website/ à partir du package R seasonal (https://cran.r-project.org/web/packages/seasonal/index.html). Voir la vignette du package pour une discussion sur les arguments par défaut de la fonction.
Editor Web API
Filters endpoint
URL: /filters
Respond the list of configured filters with their parameters.
Parameters:
-
period_start_day
(boolean): if true, returnperiod_start_day
for each series
Response example:
{
"series": {
"@frequency": "",
"period": [],
"period_start_day": [],
"value": [],
}
}
Compute endpoint
Respond a series transformed by the application of a composition of filters.
URL:
/compute?dbnomics_series_id={series_id}&filters={filters}
- where
{filters}
is like:[ { "name": "filter1", "parameters": {"param1": "value1", "param2": "value2"} }, { "name": "filter2", "parameters": {"param1": "value1"} } ]
The filters are applied in the same order they are mentioned in the URL parameter (pipeline order, not mathematical composition order).
Technical notes:
- Use OpenAPI (Swagger) to document the endpoints
- A cache (i.e. redis) SHOULD be used to prevent excessive resource usage
Editor Web UI
Home page
URL: /
Landing page
Display a form allowing to input a DBnomics series ID. Submitting the form redirects to the series edit page.
Display links to example series.
Series edit page
URL: /series?source=dbnomics&series_id={series_id}&filters={filters}&{filter_params}
Filter parameters are defined like: filters=filter1,filter2&filter1.param1=value1&filter1.param2=value2&filter2.param1=value1
URL example: /series?source=dbnomics&series_id=IMF%2FWEO%2FFRA.BCA&filters=aggregate&aggregate.frequency=annual&aggregate.method=average
Note: in the future the parameter source=
will be able to receive other values than dbnomics
, like an URL to a CSV file.
This page displays:
- a chart with the original series and its transformation, if filters are defined
- a table with the original series and its transformation (columns: period, value, new_value, ...observations)
- a list of filters where each one is a sub-form showing the filter parameters
When the filters are modified, the URL is updated and the transformed time series is updated in the chart.
Misc:
- Add a link from the "series" page of dbnomics-website to the editor (i.e. besides the "Download" button)
Questions
- is there a maximum number of filters?
- what happens if a user selects the "aggregate" filter with a frequency smaller than the original one? Should it return an error, or use the "interpolate" filter? I (@cbenz) prefer the first solution because the second one implies setting the "method" arbitrarily ("linear" or "spline").
Misc notes:
- if generic "CSV" is sent (by upload or URL), how to detect "period" and "value" columns? We should leverage datapackages (especially "time series" profile, to be invented!)