Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save historical timeseries in smaller files #4

Open
pankaj3009 opened this issue Jul 11, 2018 · 0 comments
Open

Save historical timeseries in smaller files #4

pankaj3009 opened this issue Jul 11, 2018 · 0 comments

Comments

@pankaj3009
Copy link
Owner

Problem

  1. When requesting data for a symbol, redis loads the complete Rdata file. This can consume memory if there are many large Rdata files
  2. When we try to save Rdata file after appending new data, the process for saving takes longer, especially on a spindle disk
    Both problems 1 and 2 will magnify as we move into saving smaller time resolutions and increase history.

Solution

  1. Save historical data as .rds files that serialize R object.
  2. The name of the rds object should specify the symbol and the period of the data held. If we are able to specify the default number of bars in each rds file, it is possible that we can save the rds file with symbol + starttime.
  3. loadSymbol should load a defined number of recent bars, unless a start and end date is specified. The default number of recent bars should be equal to the default number of bars expected in each .rds file.

Alternative
The current alternative is to keep data in kairosdb (the current solution). However, this has issues of speed while accessing web api for each data call. In addition there is significant maintenance and hardware costs for the solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant