Databricks Launches Apache Spark-Based Cloud Platform For Data Engineers Big Data company Databricks Inc. has launched a new version of its Apache Spark platform, which is specifically tuned for data engineers on Amazon Web Services Inc.’s (AWS). The platform, Databricks for Data Engineering is designed to assist data and machine learning engineers in creating and deploying highly optimized infrastructure for data processing on the cloud. These people are responsible for addressing business use cases like fraud detection and real-time dashboards. They also perform mission-critical operations such cleansing, transforming, and manipulating data. Databricks added that data engineering is essential for processing data to make business decisions or automate business processes using intelligent algorithms. Apache Spark is the open-source data processing engine that powers the Databricks platform. Its capabilities are greatly improved upon the MapReduce component that was introduced with Apache Hadoop. The company released a statement today stating that the new offering allows for more cost-effective data engineering with Spark and empowers data engineers to quickly and securely deploy data pipelines into production using Spark. Databricks for Data Engineering will be a complement to the company’s cloud platform and provide all enterprises with a unified platform for data analytics that facilitates seamless collaboration to accelerate data driven decisions across the organization. The company stated that the optimized platform offers:

Delilah

Performance optimization: Databricks I/O technology, (DBIO), improves processing speed with a tuned and optimized Spark version for a wide range […]

Databricks Now on AWS Databricks is the data analytics provider that created the Apache Spark analytics engine. It announced a partnership with Google Cloud to allow the deployment of its data engineering solution on another cloud platform. The San Francisco-based company is said to have “scored a trick” or won the “triple crown” by launching its solution on Amazon Web Services (AWS), Google Cloud, and Microsoft Azure. Databricks describes its solution as the “only unified data platform across all three cloud platforms.” Amazon Marketplace now offers the AWS implementation. This announcement comes after the company raised $1 billion in Series G funding. Franklin Templeton, a new investor, led the round. He joins strategic investors AWS CapitalG, Salesforce Ventures, and Salesforce Ventures. Microsoft was the lead investor in the round. Databricks users can create a “lakehouse” on Google Cloud’s elastic network, which is capable of data engineering and machine learning (ML), and analytics. Databricks now integrates to Google BigQuery’s open platform, and leverages Google Kubernetes Engine(GKE), the companies stated in a statement. This will allow its users to deploy Databricks into a fully containerized cloud environment. Databricks is built on a modern lakehouse architecture, which is cloud-based, and helps organizations eliminate the complexity and cost that are inherent in legacy data architectures,” Ali Ghodsi (CEO and co-founder of Databricks), said in a statement. “Data teams can collaborate and innovate more quickly.” This lakehouse paradigm is the driving force behind our growth and it’s exciting to see how excited our investors are about being a part. Databricks and Google Cloud have new integrations:

Delilah

Databricks is tightly integrated with Google Cloud’s analytics tools, making it easier to extend “AI driven insights” across data lakes […]

AWS Supports the Anti-Corruption Challenge for Cloud Orgs Amazon Web Services has announced its support for a new technology competition that aims to foster anti-corruption standards within the cloud computing industry. The first annual “Shield in the Cloud” competition was announced Wednesday by C5 Accelerate. This technology-focused investment firm previously partnered with AWS to launch an accelerator program for cloud startups in the United States and Europe. According to the contest website, the two companies have teamed up with SAP’s National Security Services (NS2) and nonprofit PeaceTech Lab to organize Shield in the Cloud. The goal of the competition is to find “the most innovative start–ups, corporations and not-for-profits working in cloud-based tech to promote transparency, accountability, and compliance worldwide.” The competition is open worldwide to companies and organizations that “use cloud-based technology in order to combat corruption.” The announcement states that the winners will be selected according to four categories:

Delilah

Dream Big Shield. This award is for the most innovative start-up that uses cloud to ensure anti-corruption compliance or the […]