Nobody will argue that Oracle Support is a word class experience. Like it or not, the service is here to support thousands and individuals can get lost in the process. If you navigate the process correctly, you can get results. Read more
Having the best of both worlds, ASO and BSO, doesn’t come without some gotchas. Before you jump in with both feet, beware of some things that are not supported in hybrid. As of Friday, May 22, 2020, @ISMBR in planning does NOT work. Read more
Have you ever gotten an error that reads, “Unable to retrieve variable variable name deployed in application?” When you look at the error, the variable name in the error message doesn’t exist and you aren’t trying to retrieve such a variable? Read more
I had a very interesting thing happen today that tripped me up. When loading data to a PBCS database through Planning (not as an Essbase file), I had two types of errors. I have never seen this before and this could cause some serious heartburn for administrators and possibly waste a ton of time trying to resolve. So, I am sharing for the sake of saving somebody some effort. Read more
If you have jumped into Groovy Calculations, one of the things you likely would try to do is grab a value for a sub var. Hopefully, you haven’t spent too much time before reading this. I wasted a ton of time trying to get this to work before I opened a ticket with Oracle. This class is NOT available yet and was inadvertently included in the public docs at https://docs.oracle.com/cloud/latest/epm-common/GROOV/. The development team told me they are going to remove it from the API docs. Read more
This section will talk about how to dive into debugging critical issues with Oracle EPM.
Start a Problem Log
The most useful habit to develop during issue resolution is to start a detailed log about the issue. Some problems can take days or weeks to resolve and require trying hundreds of different prospective resolution attempts. It is easy for a “small” problem to become a long winded issue. Consequently, it is hard to foresee when the issue will resemble the analogous onion: keep peeling off layers and finding more and more to fix. If the problem log is created initially, all the important details can be captured. Additionally, it is much easier to bring others up to speed (management) and create support tickets when all of the information is documented. This log should include the error as the end user sees it, the error from any logs you are able to capture, screenshots, timestamps, and things that you have tried along with the results.
Reproduce the Issue
The first thing to find out is whether the issue is reproducible. It is very difficult to solve an issue that is not reproducible. Many errors are simply ‘glitches’ and may have been caused by a very improbable event, such as a database hiccup. For instance, a database problem propagates into the Oracle EPM system, forcing it into a bad state. Such a problem may never produce itself again. Consequently, an initial step toward resolution is to restart the Oracle EPM services to bring them back into a ‘known state’. If the problem is not immediately reproducible after the restart, go back to the problem log and record everything you can. This type of issue will need to be profiled over a period of time to try and discover patterns if it occurs again.
The Numerous Logs
Once it is discovered that the issue is not a simple glitch, it is time to start digging. As mentioned previously, the first place to track down the cause of an issue is in the logs. The logs come in various forms. Here is a general breakdown of the log types:
General
| Log Type | Description |
| Windows Event Viewer | This is helpful for general system related messages. Also some modules built on Windows Technology (DCOM) will log messages here. For example, Financial Management (HFM) and Financial Data Quality Management (FDQM). |
| Application Logs | The application logs are actually generated by the Hyperion code itself. These often contain the most useful information. |
| Application Server Logs | This type of log pertains to a Java based Web Application. Most of the Hyperion modules with a web based front end have Application Server Logs. The Application Server Logs run within the WebLogic, Tomcat, or WebSphere container. |
| Web Server logs | The web server controls the handoff of web requests between the Hyperion Modules. The best way to use this log is to look for error codes (404, 401… etc) in the web log and review the corresponding URL that was used to ensure it is correct. Sometimes it might be obvious that the URL in the web log has the wrong domain, points to the wrong server, or cannot resolve the context. |
Start by reviewing the log for the product where the error is occurring. The Application Logs and Application Server Logs will be most useful at first. The goal is to find a useful error message that can be used in the next process to find a resolution to the problem.
Common Log Locations:
Unfortunately, the actual log locations change drastically between recent versions of Oracle/Hyperion products. As stated before, searching for *.log might be useful.
Example Application Server Logs:
| Essbase Admin Services | Svr2 | /Oracle/Middleware/user_projects/domains/EPMSystem/servers/EssbaseAdminServices0/logs |
| Workspace | Svr1 | /Oracle/Middleware/user_projects/domains/EPMSystem/servers/FoundationServices0/logs |
| Financial Reporting | Svr1 | /Oracle/Middleware/user_projects/domains/EPMSystem/servers/FinancialReporting0/logs |
| Analytic Provider Services | Svr2 | /Oracle/Middleware/user_projects/domains/EPMSystem/servers/AnalyticProviderServices0/logs |
| Web Analysis | Svr1 | /Oracle/Middleware/user_projects/domains/EPMSystem/servers/WebAnalysis0/logs |
Example Application Logs
| Reporting and Analysis Core | Svr3 | /Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/ReportingAnalysis/
/Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/ReportingAnalysis/stdout_console_default.log |
| Essbase | Svr4 | /Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/essbase/ |
Sifting Through the Logs:
It helps to know which modules depend on each other in order quickly pick out the respective log files to analyze. The basic idea is to determine which products are interacting and to review each log in detail for messages. It is important to review the logs of the product not only during runtime (as it is happening), but also during startup. Sometimes the fastest way to cut out the fluff is to stop the services, move or delete all the existing logs and start the environment back up. This ensures any log messages are relevant to the issue. Alternatively, one has to sift through potentially large logs looking for timestamps to ensure relevance, which can be daunting.
| Product | Depends On |
| Shared Services | Relational Database, MSAD/LDAP |
| Lifecycle Management (LCM) | Shared Services, LCM Source/Target applications |
| Essbase | Shared Services |
| Hyperion Planning | Shared Services, Essbase, Business Rules, Relational Database per App |
| Business Rules | Shared Services, Hyperion Planning, Essbase, Relational Database (single database) |
| Hyperion Financial Management | Shared Services, Relational Database (single database), DCOM (Event Viewer) |
| Financial Data Quality Management (FDM) | Shared Services, Relational Database per App, Adapters for Essbase, Planning, HFM…etc, DCOM (Event Viewer) |
| Strategic Finance | Shared Services, Relational Database (optional) |
| Data Relationship Management | Shared Services, Database Client, Adapters, DCOM (Event Viewer) |
Found an Error Message!
After discovering the error message, the first thing to ask is does this message make any sense? Try to use it within the context of your problem to solve the issue. Often, it is necessary to use external resources to resolve the issue. Use resources like Google, the Oracle Support Knowledgebase, and the Oracle Forums to further research the issue. Most often there will be information regarding your issue.
Note: If possible do not searching using end user messages, i.e. what the user sees when encountering the error. Rather, find a detailed message in the logs. The end user messages are usually very generalized and can provide misleading information because of the vast number of issues which might match the general error message.
If there is still a struggle to discover a useful error message, most of the Hyperion modules use a logging mechanism that can be changed into debug mode. The actual method will differ based on product, for instance, most modules use log4j and there is often a .properties file you can change the logging level from “ERROR” or “WARN” to “DEBUG”. For instance, to enable debugging in Hyperion Planning: log into a Planning application, go to Administration -> Manage Properties, Select the System tab, Add the property DEBUG_ENABLED with a value of True. After changing the logging level, the service will need to be restarted to reflect the changes. Turning on application debugging should provide more context clues around what the product is doing at the time and help pinpoint the error.
Nothing Found…
If these resources do not help, an Oracle Support Ticket may be required. Additionally, the Oracle Forums can be a good place to post a question. When creating a support ticket and posting to a forum, please include as much information as possible. This is where the Problem Log will come in handy.
This is a good time to look for updates and patches to the product. Check for patches and updates on http://support.oracle.com. Read the release notes for anything matching your problem. Even if there is nothing coming up, some obscure errors can be solved by simply applying a patch. Not all bugs will be in the release notes for the patch. Oracle’s hpatch process is pretty straight forward, but older environments might take some time to apply the patch. Always read the entire release notes and installation instructions before applying a patch. Also, sometimes patches are not as proven as the initial installers. This is because some patches may have just been release and only tested with a handful of clients. So ensure there is a good backup process in case the patch causes unintended problems. The Oracle hpatch process has a back out feature, but it is not always useful if the patch is half way installed and failed.
Finally, the last part of troubleshooting is intuition. As more problems are encountered and resolved, one can become more confident in resolving upcoming issues. There is no way to have encountered every issue and know the resolution, so the best that you can do is arm yourself with a good knowledge of the architecture, have a set of best practices, and lots of patience for problem solving.
This article will discuss some best practices around troubleshooting and debugging your Oracle EPM environment.
Oracle EPM (Hyperion) is a complex system that is composed many modules that work together to perform different functions. Typically, in design, creating a modular approach is a best practice. However, the Oracle EPM modules were created with disparate architectures because they acquired over time through purchasing technology from many smaller companies. As a result, the modules interact only at a high level, through network APIs. The result is a complex system with many breaking points, and error messages that can be less than informative. Consequently, troubleshooting and debugging Hyperion is an art form.
The fastest way to debug a critical problem is to have a good understanding of what components represent a known, “good” state. Only by identifying the abnormal element in the system can one start to resolve a critical issue.
Below Are Best Practices to Follow to Document a Known, Good State
What Should Be Running?
The first question you should ask is, “is everything up?” Typically you would start by checking the URLs you use for each product – ensuring you can login. These URLs would typically include the load balancing name.
Example of End User/Administrative URLs
| Shared Services | http://hyperion.svr.com/interop |
| Hyperion Planning | http://hyperion.svr.com/HyperionPlanning |
| Essbase Admin Services | http://hyperion.svr.com/easconsole/console.html |
| Workspace | http://hyperion.svr.com/workspace |
| Web Analysis | http://hyperion.svr.com/WebAnalysis |
| FDM | http://hyperion.svr.com/HyperionFDM |
Set Up Port Monitoring Software
You can also get a ton of info in a quick glance by setting up port monitoring software. Remember to include the relational database in your monitoring. Sometimes the DBA will take the database down unexpectedly, or the database user IDs will expire. Additionally, it is easy to have 20-30 Hyperion related services or processes running per environment distributed among multiple hosts. It’s too time consuming and error prone to check manually.
An Example of a Custom Java Based Port Monitor
To gather a list of ports within the environment there are a few aids to use for reference. Oracle EPM Version 11 includes a diagnostics web form that will show the status of some of the products. This can be found under the Windows Start Menu, under Programs -> Foundation. This might be a good place to get started.
Also, refer to the Oracle EPM Documentation, Install Start Here Document for typical port reference.
http://download.oracle.com/docs/cd/E17236_01/epm.1112/epm_install_start_here.pdf
An Example of Some Common JVM Ports
| Application | Server Cluster/Node(s) | Port |
| Workspace | svr01 | 45000 |
| Web Analysis | svr01 | 16000 |
| Financial Reports | svr01 | 8200 |
| Shared Services | svr01 | 58080 |
| Analytic Provider Services | svr04 | 13080 |
| Hyperion Planning | svr02 | 8300 |
| Strategic Finance | svr04 | 7750 |
| Essbase Admin Services | svr04 | 10080 |
| EPMA | svr02 | 19091 |
Stopping and Starting the Environment
Operational procedures are important too. This means having a reliable start and stop procedure for the environment. Given the complexity of the Hyperion environment, there is no excuse for problems occurring while bringing an environment back up after routine maintenance. One common issue that comes up is a service did not start completely, which can be found by using the port monitoring method above. Also, sometimes the services do not fully stop – causing residual processes to mess up the restart procedure. This might include extra ESSVR processes indicating the Essbase application did not stop properly, or a JVM process which is hung. As a precaution, check the port monitoring software after stopping the environment to ensure all the components are indeed down. It is time consuming when this type of issue is encountered. In a Windows environment you might take the environment down through your normal process, then kill any remaining “Java” and “Hyperion” related processes using task manager. In a Unix environment, after stopping the environment, “kill -9 -1” can be used as a last resort. A reboot will always solve this type of issue, though not often actually needed; it might be faster as a last resort for the unskilled admin needing to fix unexplainable issues.
Functional Validation Script
Functional validation is a set of very simple actions that can be performed within a few minutes to validate the functionality of the environment. This is absolutely necessary to perform before handing the system back over to end users after a routine maintenance. This task can be either performed by the Hyperion Admin or handed over to the Help Desk. It should, at a minimum, include running a few types reports (HFM, FR, WA), log into Essbase, Planning, HFM, etc… As you discover other reoccurring issues you may want to include more specific checks to ensure common “gotchas” are ironed out before handing over to the end users.
Knowledge of the Architecture
The Hyperion administrator has to be technologically savvy. However, many times Hyperion will tell you exactly what the problem is. The first place to look when encountering an issue is the logs! Take time to find and document the log files for every product in your environment. Most administrators are a bit intimidated by the number of logs generated by the system, but this is the first place to look. Familiarize yourself with the various logs in the environment. This includes WebLogic logs, JVM logs, Hyperion logs, and operating system logs. If you are lost, a good place to start is search for *.log, sort by date, and look at the directory for the product name. You might even keep a record of each log after starting the service successfully so that you can compare to a good state if there is a problem.
Additionally, it is necessary to have context around how the environment works. This can be achieved by reading the product manuals, looking at architecture diagrams, and attending training sessions. Get involved in the environment. Try to understand both the technical details (ports, logs, different components…etc) and the functional basics (create a planning form, edit an outline, load data into Essbase, run a consolidation) of each Oracle Module.
A Good State: Create a Knowledge Base
Hyperion issues commonly come up more than once. Create a shared, searchable, knowledgebase to track issues and their resolutions. This will help you train your team as it expands and changes over time.
Final Comments
The focus of this article was to provide the Oracle EPM Administrator with best practices on how to document the current state of the Oracle EPM System. This is the most important part of being able to understand complex issues as they come up. More technical detail will be provided in the Oracle EPM Troubleshooting and Debugging Guide (Part 2 of 2).
When using Workspace to view reports, some users have seen excessively large numbers that don’t belong. If you are having this issue, it could very well be because the default Essbase query engine in 11.1.1.x is the MDX query engine, which can cause documented bug 9062413. Essentially, this bug will cause users to see the same astronomical number in every cell that sits on an intersection to which the user does not have security access. Understandably, this can cause some concern. This issue is expected to be fixed in a future release, but until then, the query engine will need to be manually changed.
The first option is to fix the issue at a report level. This is a relatively quick process and is a good idea if you only have a handful of reports. To change the query engine setting for a particular report, follow the steps below:
1. Open the report.
2. For each grid, select the entire grid.
3. Right-click and select Data Query Optimization Settings.
4. Deselect the option “Essbase Queries Use MDX.”
5. Save the report.
Repeat the above process for each report.
For users that have a larger number of reports, a better option may be to change the query engine in the properties file on the server. The benefit to fixing the issue in the properties file is that changes only need to be made once, and all reports will reflect this change.
The file that needs to be edited is located on the Financial Reporting (app) server, typically on the path D:\<Hyperion Home>\products\biplus\lib where <Hyperion Home> represents the root location of the Hyperion install. The file that needs adjusted is the fr_global.properties file.
Open the properties file and add these lines:
# MDX Query Engine has been set as the default in Essbase 11.1.1.x. This can cause bug 9062413
# which may cause unauthorized users to see a long series of numbers in each cell when running
# reports. To solve this issue, the below line was added, which switches the query engine.
EssbaseUseMDX=false
Any line preceded by “#” is commented out. Therefore, these can say whatever you prefer, but should give anyone that views this file a good indication why this text is in the file.
Once the properties file has been updated, the following services must all be stopped in the following order, then started in the same order for the changes to take effect.
1. Hyperion Financial Reporting – Print Server
2. Hyperion Financial Reporting – Report Server
3. Hyperion Financial Reporting – Scheduler Server
4. Hyperion Financial Reporting – Web Application (Note – This service may be on the FR (Web) server, not the FR(App) server like the other three services.)
Note – This modification will apply to everyone using the server on which they are made, so be careful when making changes to a shared server.
It is possible for a database in Essbase to become corrupt. This can be caused by server hangs, software glitches, and a variety of other reasons. Although infrequent, if a database cannot be loaded for any reason, and it needs to be restored, the following actions can be a quick resolution. Keep in mind that this will remove the data and it will need to be imported from a backup export.
Before performing this, verify that the database is not attempting to recover. To determine if this is occuring, open the application log file. If it states that it is recovering free space, be patient as it may correct itself.
File Structure
Essbase has a simple file structure that it follows. It can vary with each application depending on the options used. The area to focus on for this process is below. The application and database that is being restored would take the place of appname and dbname.
Hyperion\Products\Essbase\EssbaseServer\App\AppName\DbName
Restoring To A Usable State
In this directory, files with the following extensions will need to be removed. This will delete all of the data and temporary settings that are causing the application to function improperly. It will NOT delete the database outline, calc scripts, load rules, or business rules.
- .ind (index files)
- .pag (data files)
- .esm (Essbase kernel file that manages pointers to data blocks, and contains control information that is used for database recovery)
- .tct (Essbase database transaction control file that manages all commits of data and follows and maintains all transactions)
After these files are removed, verify that the application and database is functioning. This can be done in Essbase Administration Services by starting the application. If the application doesn’t start, more research will have to be performed. If the application loads, import the most recent data backup and run an aggregation.
There are a number of other possible file types in this directory. Below is some information that may be helpful.
Audit Logs
- .alg: Spreadsheet audit historical information
- .atx: Spreadsheet audit transaction
Temporary Files
- .ddm: Temporary partitioning file
- .ddn: Temporary partitioning file
- .esn: Temporary Essbase kernel file
- .esr: Temporary database root file
- .inn: Temporary Essbase index file
- .otm: Temporary Essbase outline file
- .otn: Temporary Essbase outline file
- .oto: Temporary Essbase outline file
- .pan: Temporary Essbase database data (page) file
- .tcu: Temporary database transaction control file
Objects
- .csc: Essbase calculation script
- .mxl: MaxL script file (saved in Administration Services)
- .otl: Essbase outline file
- .rep: Essbase report script
- .rul: Essbase rules file
- .scr: Essbase ESSCMD script
Other
- .apb: Backup of application file
- .app: Application file, defining the name and location of the application and other application settings
- .arc: Archive file
- .chg: Outline synchronization change file
- .db: Database file, defining the name, location, and other database settings
- .dbb: Backup of database file
- .ddb: Partitioning definition file
- .log: Server or application log
- .lro: LRO file that is linked to a data cell
- .lst: Cascade table of contents or list of files to back up
- .ocl: Database change log
- .ocn: Incremental restructuring file
- .oco: Incremental restructuring file
- .olb: Backup of outline change log
- .olg: Outline change log
- .sel: Saved member select file
- .trg: Trigger definition file.XML (Extensible Markup Language) format
- .txt: Text file, such as a data file to load or a text document to link as a LRO used for database recovery
- .xcp: Exception error log
- .xls: Microsoft Excel file
Audit logs, or SSAUDIT, are a crucial component of backing up Hyperion Essbase applications in many environments. It is the equivalent of a transaction log in a relational database. To use this effectively, the audit log has to consistently log database changes.
If the audit feature in Hyperion Essbase is used, the following information is absolutely critical to know to effectively manage this feature. If the application is on a shared environment where multiple groups/people are administering the applications, it is critical that everybody understands this, and plays nicely together!
The audit logs are turned off without any notification when the following actions occur on an Essbase server. To turn the audit feature back on, the Essbase application in question has to be stopped and started. It is not required to cycle the Essbase service.
- Any operation that causes a database restructure.
- The creation of a new application
- The creation a new database
- Copying a database
- Renaming a database
After any of these operations occurs on the server, stop and start all applications that use the audit feature.