Skip to main content
T
TT8
Newcomer
September 30, 2024

Source Data Condition check before Import

  • September 30, 2024
  • 2 replies
  • 0 views

Is there a way I can check my source records before importing data into a stage? If I am in a workflow for Import, Validate and Load, Even before the data is imported, I would like to check my source record for a condition like if it is less than a particular value. Only if the condition satisfies, I need to proceed to import else stop the workflow.

If someone has done it, can you share the setup and code for the same.

Appreciate your help!

2 replies

L
OneStream Employee
ludodepazOneStream Employee
OneStream Employee
October 1, 2024

Hi, you can write a Parser Rule, it will run when you Import data. You create your rule and you call it in the Data Source like this: 

ludodepaz_0-1727790862028.png

T
TT8Author
Newcomer
October 1, 2024

Thanks! I will try it. Is there a way that we can also write error email function in parser rule or call an extender rule from parser rule? I need to send an error email if source condition does not satisfy. Would really appreciate your help

L
OneStream Employee
ludodepazOneStream Employee
OneStream Employee
October 1, 2024

Yes, you can send an email. I see 2 options

  1. if you execute the Import from a DM job and you're on V8.2 or later, you can send an email from the Data Management Sequence.
  2. There is another option, which is very slick I believe, you can log the errors in a file and then send the file by email and with a little bit of code (like add a boolean with 'has Errors'), you can send the email only if there is an error and send the details. 

I posted a blog on how to log with a file and send the file via email, I hope it helps!
Log like a champ with OneStream - OneStream Community (onestreamsoftware.com)

J
OneStream Employee
JoakimKOneStream Employee
OneStream Employee
October 3, 2024

Is this a file load or a direct connect import?

It sounds to me that you are asking for a check before import to evaluate if it should even bother with the import at all

So, if using a file based import a parser rule is good for such a thing, a parser rule is a rule that will run on a record by record basis and is already part of the import, so your import would have already started when it runs, and it will evaluate each record on its own merit. This works fine if the evaluation you want to perform is on the first line, but for anything else it will likely import some items and some not using this method.

If you are using a direct connect method, you can evaluate the dataset you are wanting to import before even initiating the import at all, and you would need to do this in the connector business rule. Alternatively, if its connector based you could even make it a requirement for the data to even be returned at all if the criteria is not satisfied (saving the time needed to even transfer any data in the first place) by using a where clause (for SQL) or parameter if using an API.

Joakim

    Terms & ConditionsAccessibility statement