The Ultimate Big Data Architecture Checklist

The joy of working as a Customer Success Solution Architect is that I have the opportunity to work with many different customers and each challenges us with a different Big Data use case.

I’ve worked with enterprises that offload their Netezza database into the cloud. I’ve seen companies analyze social media data in real-time. I’ve helped teams streamline operational processes and increase efficiency in production lines. Big Data provides enterprises a competitive advantage and reduces operational costs across a these varied scenarios. However, setting up a big data environment is not for the faint-hearted – or is it?

Sticking to Principles…

AtScale helps enterprises to get the most from their data lake. Drawing from our experience working with different enterprises and their different levels of Big Data maturity we introduced the data architecture principles:

• Treat data as a shared asset
• Provide the right consumption interfaces
• Ensure security and access
• Have one common vocabulary
• Drive information through data stewardship
• Eliminate data copying and movement

The goal of implementing Big Data can be a daunting task; however, when properly executed, Big Data has the potential to transform an organization’s ability to manage structured and unstructured data at scale.

Real Use Cases

A leader in the IoT industry wants to evaluate performance of their new energy efficient thermostat. To evaluate the results, they must collect several distinct data points, ranging from weather data, indoor and outdoor humidity levels, and air conditioning or furnace run time. This data comes in different formats and can also be collected in real time.

In the past, they stored this data in multiple traditional relational databases. It was then aggregated on each machine and transferred to a centralized location. This ETL process was costly and time consuming.

Changes that Pay Off

To reduce the latency and labor involved, they deployed a Hadoop environment. With these changes, all of the instrument data is now collected to a single location. Implementing a modern data architecture eliminated the need for data movement, allowed them to analyze all their data, and now their team is able to generate analytics and reports in near real time.

How Can You Succeed?

To help you build your next Big Data environment, here is the ultimate checklist that will help you succeed while avoiding the most common mistakes:

• Break down success metrics into stages (i.e. clearly defined use cases)
• Identify a semantic layer for the business users (common vocabulary across data)
• Identify end-to-end needs and solutions (data ingestion, data wrangling, and BI tools and data consumption)
• Collaborate on SLAs (define your querying time SLA and your data prepping SLA).
• Determine resources to achieve your SLAs (your system is only going to be as fast as the slowest node in the cluster)
• Identify a security component of data access (e.g., onboarding a business analyst)
• Identify a key sponsor and expert resources (you need people who will be actively using the system)

We understand that each enterprise may be at a different stage of their big data journey. Based on our experience, we can to provide guidance on your big data architecture. Feel free to reach out to our team with your big data analytics concerns.

AtScale will give you all the flexibility and capacity you need to expand your next big data project.

See AtScale Intelligence Platform in action, Sign up for this webinar!
We invite you to learn more about AtScale today!