Dzone-2018年大数据技术指南（英文版）-2018-38页-6mb

2018 DZone Guide to Big Data: Summary

Core Content Overview

The 2018 DZone Guide to Big Data provides an in-depth look at the evolving landscape of Big Data technologies, focusing on stream processing, statistics, and scalability. It highlights the challenges and opportunities associated with managing and analyzing large volumes of data, including structured and unstructured data, fast-moving and historical data, and the role of emerging technologies like blockchain, cloud computing, and IoT in shaping the future of Big Data. The guide also emphasizes the importance of data quality, analytics project success, and the growing popularity of Python over R in data science.

Main Points and Key Insights

1. Big Data Definition and Challenges

• Big Data is defined by the three Vs: Volume, Velocity, and Variety.
• Traditional databases are not sufficient for handling Big Data due to its complexity.
• Data contextualization, validity, noise, and abnormality are significant challenges.
• Volume is the most common issue, with 76% of respondents dealing with large data sets.
• Velocity and Variety also pose challenges, with 46% and 45% of respondents citing these as problems.

2. Data Storage and Cloud Adoption

• Cloud storage is becoming more popular, with 39% of respondents using it compared to 33% on-premise and 23% using hybrid solutions.
• Amazon Web Services (AWS) is the most preferred cloud vendor, used by 70% of respondents, followed by Google Cloud (57%) and Microsoft Azure (39%).
• The use of cloud solutions for Big Data increased by 8% in 2018, while on-premise and hybrid solutions decreased by 6% and 4%, respectively.
• Python has overtaken R as the dominant language for data science, with 70% of respondents using Python versus 50% using R.
• Kafka is widely used for real-time data processing, with 61% of respondents using it.
• MongoDB is the most common NoSQL database used for Big Data, with 29% of respondents selecting it, compared to 11% more than the next closest database.

3. Blockchain and Big Data Integration

• Blockchain is being explored as a tool to enhance data security, transaction integrity, and streamlining workflows.
• It enables peer-to-peer data storage and smart contracts, which automate agreements and reduce reliance on middlemen.
• Use cases include agriculture, where blockchain can track raw materials, product quality, and compliance across supply chains.
• HPCC Systems is highlighted as an open-source platform that supports Big Data analytics, ETL, and real-time processing.

4. Data Integration and Analytics Success

• Data integration is critical for data-driven companies, enabling efficient data movement and accessibility.
• Stitch Fix is mentioned as a case study where data integration is used to manage customer data, inventory, and supply chain data.
• Data quality remains a major concern, with 64% of respondents citing unsanitized or unclean data as a challenge.
• Data analytics projects often fail due to unclean data, tight timelines, and limited talent.
• A checklist of ten tips is provided to ensure project success, emphasizing the need for data preparation, real-time processing, and cloud integration.

Key Recommendations

• Use Apache Kafka for real-time data processing.
• Employ document store databases like MongoDB for handling semi-structured data.
• Cloud storage is recommended for smaller enterprises due to lower upfront investment, but hybrid or on-premise solutions may be better for sensitive data or fast analytics.
• Data sanitation is crucial to ensure accurate insights and project success.
• Python is increasingly preferred over R in data science due to its flexibility, libraries, and ease of use.
• Blockchain can be used to automate transactions, improve data security, and enable smart contracts.
• HPCC Systems is a massively scalable, open-source platform that simplifies Big Data management, ETL, and predictive modeling.

Conclusion

The 2018 DZone Guide to Big Data outlines the evolving nature of Big Data technologies and their growing importance in modern business. It emphasizes the need for effective data management, cloud adoption, data quality, and innovative tools like blockchain and HPCC Systems to harness the full potential of Big Data. The guide serves as a valuable resource for understanding the current state of Big Data and how to leverage it for business success.