Dealing with small things that can have a bigger impact
Creating A Robust Infr structure To Utilize
Creating A Robust Infrastructure To Utilize Data Streams
The first step is to collect the data from where it is produced, it
could be the recording of client calls, data gathered from
instrumentation, sensors, logs, or external systems.
It is raining heavily in my city. It reminds me of the data that is
produced or recorded by even a moderately sized organization.
Numerous data streams are flooding the business every day. In order
to harness this force, the business has to judiciously create the data
lakes and dam the data without harming the business environment,
using the data gathered to gain insights, predictions, and directions
from it. Ok, let me stop this analogy before we head towards a cloud
A smart organization builds a robust infrastructure to utilize this
Creating A Robust Infrastructure To Utilize
Get more insights on our website
The second step is to transform and store this data in a usable
format, be it structured or unstructured. Start cleaning and
Prepping the data. This is done by the data engineering team
The third step
is to aggregate and
label the data.
Experimentation, A/B testing, Training data, and using ML
algorithms. This task is done by data analysts and data
The instrumentation and infrastructure engineers would be
working at this stage
Each one of these steps requires a different type of training and
expertise. The boundaries between the steps are porous and the
people working on them have overlapping skills. So it is imperative
that the correct person is assigned the correct role. Not realizing
this can result in unhappy scientists and underutilization of the
It is best to match the steps and skills correctly.