The conversation on Slack revolved around pulling data from custom data indexes via API. It was revealed that while there are 7 indexes with varying numbers of items, only 10,000 rows can be queried from each one. This is because EdgeDX expects filtering and aggregation to be done on the tenant side rather than the client side. The logic behind data retention was also brought up, with the max amount of data being 30 days. The discussion ended with the advice to query the collapsed index for more recent data. The full conversation can be found at https://controlup.slack.com/archives/CAZ3VBF6B/p1617709089046900.
Read the entire ‘Discussing Data Retrieval from Custom Data Indexes at ControlUp’ thread below:
We are looking at pulling data via API from a custom data index and I’m trying to understand how much data is stored in the custom data indexes. When I look in edge DX I see 7 indexes each with a varying amount of "# items" and size but when I query each index I get exactly 10,000 rows. I also saw if I generate a report and export that as a CSV I get somewhere in the 65,000 range.
I’m not sure this is an issue exactly I guess I’m mostly trying to understand how these indexes iterate and how much data I should expect to find in them.
10k is by design. EdgeDX expects that you filter, aggregate or query on the tenant side rather than fetch the entire index and filter on the client side.
For example, instead of retrieving all devices (/api/data/_devices) then grouping by hw_model. You’re should instead ask the tenant to group for you:
Do note that you’ll have to URL encode the url. But Powershell will do that for you. But if you use something else you might have to URL encode yourself
For example the URL encoded version of my example above:
Thanks Dennis, though I’m more wondering around what the logic is for data retention? Is it just a max of 70k records per custom table? Also the data indexes seem to increment by 100 aka Login detail 20240100, 20240200 etc. I would have assumed the newest index wouldn’t already have 10,000 entries in it but they always seem to have that.
is it better to just ignore the indexes and just submit as query if I want only recent data?
You should be able to query the collapsed index. For example, I don’t query device_status_20235000 or device_status_20235100. I just query device_status and let the tenant deal with which actual index to query.
Though the last part of my question is how much data do the custom data sets hold? it seems they all have 10k items and they create a total of 7 indexes. Does that mean they hold a max of 70k records?
they aren’t limited by records. They are limited by time. 30 days. Though some might be a bit longer. 30 days is what the Edge team commits to
Continue reading and comment on the thread ‘Discussing Data Retrieval from Custom Data Indexes at ControlUp’. Not a member? Join Here!
Categories: All Archives, ControlUp Edge DX