PBCS Data Map / Smart Push Has Data volume Limits
Introduction
When moving data in PBCS with Data Maps or Smart Pushes, they have limits on how much data can be moved. The amount of data can be seen in the logs, and look something like this.
Failure
Exporting data…
Exported data file(s) size is: 207.1 MB.
Push Data failed. Error: Exported data size of data map that is being executed from groovy is more than permissible amount: 100 MB.
Success
Exported data file(s) size is: 464.7 MB.
EXPORT elapsed time: 39584
IMPORTING – AppName: AreakFin
TRANSFORM elapsed time: 63634
IMPORTING elapsed time: 21166
TOTAL elapsed time: 124553
Prior to the Feb, 2018 release, the following did not always hold true. If you are/were seeing inconsistencies, see Bug Report: Push Data failed. It also includes information about how the data cap works, as it is different between Data Maps and Smart Pushes, which is worth reading.
Data Movement Limits Identified
I got the following information from Oracle, and it is useful if you are using the data movement functionality. When these are developed, it is a good idea to evaluate the size and plan for growth. If the production data movements are nearing the thresholds, it is recommended to be proactive and try to reduce the POV that is used to move the data. If it can’t be reduced, one option is to split it into multiple pushes which can be done with Smart Pushes on the Data Form save, or with Groovy. Groovy also allows you to further condense the POV by dynamically changing the POV based on the cells edited, which is the most productive and efficient way to handle these.
So, here is what was documented. The data limits imposed on the movement methods are below.
- There is not a cap when running a Data Map
- When executing the following, there is a cap of 100MB
- Smart Push on a Data Form
- Smart Push via a Groovy Calculation
- Data Map via a Groovy Calculation
Summary
If you are not seeing this, I would recommend opening a ticket with Oracle to resolve. I will be writing a post explaining how to execute and override POVs in Smart Pushes and Data Maps with a Groovy Calculation in the near future, so look for an article in my Adventures in Groovy series.
Hi Kyle,
I found your blog is very helpful. Thank you for writing and sharing your knowledge.
I hit the limit that you mentioned when running data map process via groovy. Apart from the obvious workaround by restricting the slice I’m pushing, do you have any other method? Based on your experience, is using grid builder to push data better than using data map via groovy?
Thank you
Regards,
Vilona
I don’t know if it is globally better or not. I have found instances where I prefer it. One is when you hit the limit. With the gridbuilder, you have options. You can only submit what has changed, for example, and that has overcome any form save I have ever had where I hit the limits. In instances where I did hit the limit, I was able to iterate through slices. I am working on a blog to explain this actually, but assume product is large. What I have done is iterate through all the products, x number at a time. So do 100 products, then the next 100 products, etc.
You can also split the push up and just do multiple pushes in Groovy similar to what I explained above. On several forms I have done several pushes – split them by local and USD, plit them by groups of accounts like Income or Balance Sheet, and the like. Hope this helps! Thanks for you kind words as well, I appreciate it.
Hello Kyle,
This article is very helpful, thank you for having sharring it.
Is there any new update or new feature to get around this problem?
Thank you,
Best regard
Ismail
There are no changes to this other than I heard from product dev that they are going to increase the limits. I write groovy calculations to iterate through chunks of the volume. For example, assume you have a form with multiple products. If the 5 products is under the limit, I iterate through 5 products at a time. I also normally use the REST API to execute the maps so that I can run more than one at a time. With REST, you can run 3 at a time, monitor them, and as soon as one finishes, initiate another. I am writing a class on this now that might benefit you. It will be available at in2hyperion.podia.com.