Skip to main content
N
NicoleBruno
Veteran
November 14, 2023

What's the best way to pull 16M+ rows of data

  • November 14, 2023
  • 1 reply
  • 0 views

Hello! 
I have a user group who needs a specific set of data pulled just after close every month in order to populate other reporting tools they use. Here's the formatting they require: 

NicoleBruno_0-1699992387253.png

I'm struggling on my options because this results in about 16M+ rows of data. I get an error when trying to export the CV and a QV won't show all rows even with the max # of seconds set to the largest number (which I believe is 600). A DM extract will not show the right format but I don't think a QV can be auto run and sent via Parcel Service. What are my other options? 
Thanks for helping me brainstorm!

1 reply

J
Jorge_Haces
Newcomer
November 14, 2023

You can use this function "BRApi.Import.Data.FdxExecuteCubeView(si, cubeViewToExtract, entityDimName, entityMemFilter, scenarioDimName, scenarioMemFilter, timeMemFilter, nameValuePairs, includeCellText, useStandardFields, filter, parallelQueryCount, logStats)" to retrieve the data into a DataTable. Then, using a StreamWriter to a CSV.

Hope it helps!

 

    N
    NicoleBrunoAuthor
    Veteran
    November 14, 2023

    Thanks - is there more instruction somewhere that you know of which I could review to see if this will allow the flexibility I need? 

    J
    OneStream Employee
    JackLacavaOneStream Employee
    OneStream Employee
    November 15, 2023

    It's part of the so-called FDX (Fast Data eXtract) APIs. If you search around the forum you should find quite a few examples. We also cover them in the BiBlend class, so if you know anyone who attended that you can ask him for the material. They are in the latest docs at https://documentation.onestream.com - they show them applied to Data Source Connectors, but will work as well in Dashboard Data Set rules or any other type of rule.

    Basically, they allow fast extraction of Cube Views, Data Unit records, or Stage records, and also allow to pivot them on the Time dimension if necessary (i.e. getting a column for January, one for February, etc, which can compress the number of records pretty dramatically). I don't think flexibility will be a problem, but they do require a bit of coding to set up all the parameters they require.

      Terms & ConditionsAccessibility statement