Dear SAP BW enthusiasts!
Recently, one of my customers asked me for possible approaches to implement real-time reporting. During preparation my research also covered “Streaming Process Chains” with surprising results. For this reason I thought it would be a good idea to share “one of the best kept secrets of BW functionality” here in the SAP community.
This blog post will focus on the streaming process chain run modes and will provide URLs to additional sources in the end. My blog post uses the general term “SAP BW” which refers to both SAP BW 7.5 and SAP BW/4HANA.
In general, acquiring data as fast as possible should not be regarded as first choice. Instead, SAP recommends leveraging virtualization and data federation wherever possible. SAP BW on HANA and SAP BW/4HANA provide multiple suitable modeling options, mainly based on SAP HANA integration services (SDA, SDI), SAP HANA modeling options (Calculation Views), and Open ODS Views or CompositeProviders as virtual InfoProviders on the BW application side.
Only if these approaches do not meet your business requirements, loading data into SAP BW should be evaluated as a serious alternative. In the past, “Real-Time Data Acquisition (RDA)” was the concept available for InfoCubes and DSOs (classic). However, with the introduction of ADSOs, SAP decided to retire RDA and to enhance the existing framework of process chains instead.
Streaming is an operation mode of BW process chains which can be set in the properties of each process chain. When a process chain is running, each step takes a certain amount of time; the interesting question is what actually happens, when the chain is started again (maybe even several times) during this period, and the execution reaches the process that is already active from a previous execution:
In a nutshell, the Streaming mode explained above facilitates the data transfer from a source system to SAP BW in almost real-time. Compared to replication concepts like SAP SLT with a latency of milliseconds to seconds, here the realistic latency is rather counted in minutes in the best case.
In general, there are different run modes to be distinguished: A Push and two Pull run modes. The main difference between them is whether the consuming system or the source system is triggering the data transfer.
A) The PUSH Run Mode
Main Characteristics:
A trigger is received from a so-called ODP Daemon in the source system if a new record is generated. This event starts the immediate execution of the process chain which transfers the new data to the SAP BW InfoProvider.
This is the preferred mode for ODP sources, because the process chain runs only when new data is available in the source. The process chain is scheduled once in streaming mode, the definition of periodic repetitions is not required.
The daemon has to be scheduled as regular job ODQ_DAEMON_CLIENT_xxx in the ODP Queue (tr. ODQMON, GoTo / Real-Time Daemon…) or by tr. ODQ_DAEMON.
Preconditions:
B) The Intelligent PULL Run Mode
Main Characteristics:
SAP BW checks very frequently (up to every 60 seconds) if a new record is generated in the source. If this check is successful, the process chain is started immediately to transfer the new data to the target InfoProvider. The process chain is scheduled once in streaming mode, the definition of periodic repetitions is not required.
This is the preferred mode for HANA sources, because the process chain runs only when new data is available in the source system.
Preconditions:
C) The Scheduled PULL Run Mode
Main Characteristics:
SAP BW executes a process chain which leverages the streaming framework, no matter if there are new data in the source or not. The Streaming process chain is scheduled as a periodic job and it can be executed very frequently if required (up to each minute). If there are updates available, they will be loaded immediately to SAP BW. On the other hand, if there is no new data, then an empty request is created in the target InfoProvider.
Preconditions:
Summary
The Streaming property is a really cool enhancement of the process chain concept in SAP BW. It provides you a flexible toolset, if you really have to load data as fast as possible into a BW InfoProvider. The preferred run modes are the PUSH mode and the Intelligent PULL mode, as they only trigger the DTP execution if new data is generated in the source system.
I recommend following SAP Education offerings if you want to learn more about this topic and much much more. We have prepared a sophisticated scenario where you can explore streaming process chains hands-on. Both trainings are currently upgraded to SAP BW/4HANA 2.0 SP07 with more details about streaming process chains. Availability is expected by mid-2021.
References
For additional details refer to the SAP help portal and following SAP notes:
There are also some additional blog posts in this community which provide interesting use cases for the run modes introduced above:
Okumaya devam et...
Recently, one of my customers asked me for possible approaches to implement real-time reporting. During preparation my research also covered “Streaming Process Chains” with surprising results. For this reason I thought it would be a good idea to share “one of the best kept secrets of BW functionality” here in the SAP community.
This blog post will focus on the streaming process chain run modes and will provide URLs to additional sources in the end. My blog post uses the general term “SAP BW” which refers to both SAP BW 7.5 and SAP BW/4HANA.
In general, acquiring data as fast as possible should not be regarded as first choice. Instead, SAP recommends leveraging virtualization and data federation wherever possible. SAP BW on HANA and SAP BW/4HANA provide multiple suitable modeling options, mainly based on SAP HANA integration services (SDA, SDI), SAP HANA modeling options (Calculation Views), and Open ODS Views or CompositeProviders as virtual InfoProviders on the BW application side.
Only if these approaches do not meet your business requirements, loading data into SAP BW should be evaluated as a serious alternative. In the past, “Real-Time Data Acquisition (RDA)” was the concept available for InfoCubes and DSOs (classic). However, with the introduction of ADSOs, SAP decided to retire RDA and to enhance the existing framework of process chains instead.
Streaming is an operation mode of BW process chains which can be set in the properties of each process chain. When a process chain is running, each step takes a certain amount of time; the interesting question is what actually happens, when the chain is started again (maybe even several times) during this period, and the execution reaches the process that is already active from a previous execution:
- A classic process chain is scheduled in the background. Then the second execution waits 10 minutes for the previous execution to complete. If the previous run is not completed within this period, the second execution is canceled, and the process chain is interrupted and fails with an error.
- A process chain in streaming mode is executed using a new Queued Task Manager Framework. This enables a flexible number of “worker” jobs that get the next process from a queue. If a second, third, or further execution reaches the current active process, the request to start this process once more is written to a queue only. No more resources are used, and the system does not wait. As soon as the current process is finished, the worker job receives all of these start requests for a given process and starts the process just once. The process chain framework takes the last chain execution in the queue and assigns the process execution to it. This means that the second and third chain executions are dropped, and only the most recent chain execution is continued.
In a nutshell, the Streaming mode explained above facilitates the data transfer from a source system to SAP BW in almost real-time. Compared to replication concepts like SAP SLT with a latency of milliseconds to seconds, here the realistic latency is rather counted in minutes in the best case.
In general, there are different run modes to be distinguished: A Push and two Pull run modes. The main difference between them is whether the consuming system or the source system is triggering the data transfer.
A) The PUSH Run Mode
Main Characteristics:
A trigger is received from a so-called ODP Daemon in the source system if a new record is generated. This event starts the immediate execution of the process chain which transfers the new data to the SAP BW InfoProvider.
This is the preferred mode for ODP sources, because the process chain runs only when new data is available in the source. The process chain is scheduled once in streaming mode, the definition of periodic repetitions is not required.
The daemon has to be scheduled as regular job ODQ_DAEMON_CLIENT_xxx in the ODP Queue (tr. ODQMON, GoTo / Real-Time Daemon…) or by tr. ODQ_DAEMON.
Preconditions:
- This mode is only available for ODP DataSources of types SAP Extractors, ABAP CDS Views, BW InfoProviders and SLT.
- Another precondition is that DataSources are Delta-capable:
- ODP_SAP or ODP_CDS: The delta-capability is dependent on the implementation.
- ODP_BW: The delta-capability depends on the source InfoProvider. Supported sources are DSOs (classic), InfoCubes and ADSOs.
- ODP_SLT: all DataSources are delta-capable in general.
- Finally, the DataSources need to support Streaming:
- ODP_CDS, ODP_BW, ODP_SLT: all delta-capable DataSources support this property by default.
- ODP_SAP: This property depends on the individual implementation of the extractor. For example, LO Cockpit DataSources (2LIS_*) are generally not streaming-enabled due to their complexity (e.g. due to the queued delta), but other SAP delivered DataSources might support this feature. However, most of the SAP delivered DataSources are not streaming-enabled in the delivered version. There are some corrections provided (see SAP notes 1692484, 1780297), but in most of the cases so-called Realtime-flag has to be set manually in the SAP source system (e.g. by a customer report updating ROOSOURCE-REALTIME to value ‘X’). However, consider that this makes only sense for DataSources which are able to provide new data in real time, which means a delta is available at every point of time (see also SAP note 905089, not released currently and so SAP internal only).
Customer developed DataSources do not have this property initially as well. It must be enabled explicitly in the DataSource definition (Tr. RSO2). The corresponding property is called Real-Time enabled and is available in the customizing settings for the Delta mode.
B) The Intelligent PULL Run Mode
Main Characteristics:
SAP BW checks very frequently (up to every 60 seconds) if a new record is generated in the source. If this check is successful, the process chain is started immediately to transfer the new data to the target InfoProvider. The process chain is scheduled once in streaming mode, the definition of periodic repetitions is not required.
This is the preferred mode for HANA sources, because the process chain runs only when new data is available in the source system.
Preconditions:
- This mode is available for HANA DataSources only.
- Another precondition is that the DataSources are Delta-capable.
- Finally, the DataSources need to support Streaming, which is controlled by the type of remote connection:
- SAP HANA Smart Data Access (SDA): Not enabled for streaming in general
- SAP HANA Smart Data Integration (SDI): Streaming is available for adapters which support “Real-time Change Data Capture” as documented in the PAM functional overview of SAP HANA SDI 2.0. Currently those are: ASEAdapter, ASEECCAdapter, DB2ECCAdapter, DB2ECCLogReaderAdapter, MssqlECCAdapter, MssqlLogReaderAdapter, OracleECCAdapter, OracleLogReaderAdapter, PostgresSQLLogReaderAdapter, HanaAdapter, TeradataAdapter, CloudDataIntegrationAdapter and TwitterAdapter.
C) The Scheduled PULL Run Mode
SAP BW executes a process chain which leverages the streaming framework, no matter if there are new data in the source or not. The Streaming process chain is scheduled as a periodic job and it can be executed very frequently if required (up to each minute). If there are updates available, they will be loaded immediately to SAP BW. On the other hand, if there is no new data, then an empty request is created in the target InfoProvider.
Preconditions:
- This mode is available for ODP DataSources and SAP HANA DataSources which are not streaming-enabled.
Note: The fact that Streaming is not enabled for a DataSource as a property does not imply, that you cannot run the process chain in streaming mode. You can still leverage the streaming framework to run process chains with high frequency. The DataSource setting ‘Streaming cannot be triggered‘ just means that the process chain needs to be scheduled to pull data from the source (in up one minute intervals), instead of the process chain being started (triggered) when new source data is available. - Another precondition is that the DataSource is Delta-capable.
Summary
The Streaming property is a really cool enhancement of the process chain concept in SAP BW. It provides you a flexible toolset, if you really have to load data as fast as possible into a BW InfoProvider. The preferred run modes are the PUSH mode and the Intelligent PULL mode, as they only trigger the DTP execution if new data is generated in the source system.
I recommend following SAP Education offerings if you want to learn more about this topic and much much more. We have prepared a sophisticated scenario where you can explore streaming process chains hands-on. Both trainings are currently upgraded to SAP BW/4HANA 2.0 SP07 with more details about streaming process chains. Availability is expected by mid-2021.
References
For additional details refer to the SAP help portal and following SAP notes:
- SAP BW/4HANA help about Process Chains for Streaming
- SAP note 2844555: Note List for Process Chain Streaming Corrections (BW Note Analyzer)
- SAP note 2372534: Streaming data from local SAP HANA schema with generic BW delta
- SAP note 2447916: How to transfer Real-Time Data Acquisition (RDA) to Streaming Process Chains in BW/4HANA
There are also some additional blog posts in this community which provide interesting use cases for the run modes introduced above:
- Streaming Process Chains – Scheduled PULL via ODP_SAP and PUSH via ODP_SLT
Streaming Process Chains | SAP Blogs - Streaming Process Chains – PUSH via ODP_CDS
Delta Extraction and Real Time Streaming ODP-CDS to BW | SAP Blogs - Streaming Process Chains – PUSH via ODP_SLT (1)
https://blogs.sap.com/2017/09/26/slt-odp-real-time-data-replication-from-sapsource-to-bw-adso/ - Streaming Process Chains – PUSH via ODP_SLT (2)
https://blogs.sap.com/2017/10/13/slt-odp-real-time-data-replication-from-nonsapmssql-to-bw-adso/
Okumaya devam et...