Can we do it? Yes we can!
On a recent project, the client posed an interesting challenge. Requirements as follows:
- We want a Process API that can fetch data from multiple systems
- Data will come from different sources; Relational Databases (Oracle, MSSQL & MySQL), CSV Files, and external APIs
- The API must have the ability to merge data from different data sources, regardless of source Data Technology or Data Type
- We want the ability to add additional Data sources at Run-time using only configuration; no code-changes
… and then the clincher! The Process API will have no knowledge of any underlying data structures at Design-Time.
In other words, the process API will be given a report structure and a list of Data Sources designed by an end-user at Runtime. From that, it needs to figure out:
- Which Data Sources to query
- The underlying data structure
- Which fields to merge on
- The type of joins to perform and finally
- The output structure of the JSON
… all of which will vary on a case-by-case basis at run-time — and this will be fully user-driven.
I won’t bore you with the long list of technical challenges we needed to overcome in order to achieve this.
In short, the client asked, ‘Can we do this?’
We answered, ‘Yes we can!’
We did!