Databricks open-sources Delta Lake to make data lakes more reliable

Databricks, the company founded by the original developers of the Apache Spark big data analytics engine, today announced that it has open-sourced Delta Lake, a storage layer that makes it easier to ensure data integrity as new data flows into an enterprise’s data lake by bringing ACID transactions to these vast data repositories.

Delta Lake, which has long been a proprietary part of Databrick’s offering, is already in production use by companies like Viacom, Edmunds, Riot Games and McGraw Hill.

The tool provides the ability to enforce specific schemas (which can be changed as necessary), to create snapshots and to ingest streaming data or backfill the lake as a batch job. Delta Lake also uses the Spark engine to handle the metadata of the data lake (which by itself is often a big data problem). Over time, Databricks also plans to add an audit trail, among other things.

“Today nearly every company has a data lake they are trying to gain insights from, but data lakes have proven to lack data reliability. Delta Lake has eliminated these challenges for hundreds of enterprises. By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’,” said Ali Ghodsi, co-founder and CEO at Databricks.

What’s important to note here is that Delta lake runs on top of existing data lakes and is compatible with the Apache spark APIs.

The company is still looking at how the project will be governed in the future. “We are still exploring different models of open source project governance, but the GitHub model is well understood and presents a good trade-off between the ability to accept contributions and governance overhead,” Ghodsi said. “One thing we know for sure is we want to foster a vibrant community, as we see this as a critical piece of technology for increasing data reliability on data lakes. This is why we chose to go with a permissive open source license model: Apache License v2, same license that Apache Spark uses.”

To invite this community, Databricks plans to take outside contributions, just like the Spark project.

“We want Delta Lake technology to be used everywhere on-prem and in the cloud by small and large enterprises,” said Ghodsi. “This approach is the fastest way to build something that can become a standard by having the community provide direction and contribute to the development efforts.” That’s also why the company decided against a Commons Clause licenses that some open-source companies now use to prevent others (and especially large clouds) from using their open source tools in their own commercial SaaS offerings. “We believe the Commons Clause license is restrictive and will discourage adoption. Our primary goal with Delta Lake is to drive adoption on-prem as well as in the cloud.”

Managed By Q launches a new task management feature for office managers

Managed By Q, the office management platform recently acquired by WeWork, has today announced the launch of Task Management.

The feature comes to Managed By Q by way of Hivy, a startup acquired by MBQ back in 2017, that focuses on connecting a company’s employees to the office manager that handles their requests.

Pre-Hivy, collecting requests and tracking projects across a large number of employees was a tedious, fragmented process. Hivy created a dashboard that organizes all those requests in a single place.

Since the acquisition, Managed By Q and Hivy have been working to integrate their respective platforms. Where Managed By Q connects office managers to the right vendor or MBQ operator to handle the job, the new Task Management system will connect office managers with the employees making the requests in the first place, essentially putting the entire pipeline in a single place.

Obviously, the path to full integration was a long one.

“What I think matters most,” said Hivy cofounder Pauline Tordeur, speaking about the process of intertwining two separate products, “is that we knew why we were doing this and what the future would look like when we integrate. Having this vision and outlook from the very beginning is important.”

The timing is interesting in that this is the first product announcement Managed By Q has made since it was acquired by WeWork.

“It’s hard to describe the feeling,” said Managed By Q cofounder and CEO Dan Teran of being acquired by The We Company. “There is a perception of WeWork from the outside, but since I’ve been spending a lot of time getting to learn the business firsthand, I think there is just so much potential.”

He noted that Managed By Q is indeed setting out to do with WeWork what it just completed with Hivy.

“We set out to build the operating system for space, and one of the biggest things we missed is the space itself,” said Teran. “That’s actually the hardest part for most people. So now that becomes another ingredient we can deliver to our customers.”

Docker developers can now build Arm containers on their desktops

Docker and Arm today announced a major new partnership that will see the two companies collaborate in bringing improved support for the Arm platform to Docker’s tools.

The main idea here is to make it easy for Docker developers to build their applications for the Arm platform right from their x86 desktops and then deploy them to the cloud (including the Arm-based AWS EC2 A1 instances), edge and IoT devices. Developers will be able to build their containers for Arm just like they do today, without the need for any cross-compliation.

This new capability, which will work for applications written in Javascript/Node.js, Python, Java, C++, Ruby, .NET core, Go, Rust and PHP, will become available as a tech preview next week, when Docker hosts its annual North American developer conference in San Francisco.

Typically, developers would have to build the containers they want to run on the Arm platform on an Arm-based server. With this system, which is the first result of this new partnership, Docker essentially emulates an Arm chip on the PC for building these images.

“Overnight, the 2 million Docker developers that are out there can use the Docker commands they already know and become Arm developers,” Docker EVP of Business Development David Messina told me. “Docker, just like we’ve done many times over, has simplified and streamlined processes and made them simpler and accessible to developers. And in this case, we’re making x86 developers on their laptops Arm developers overnight.”

Given that cloud-based Arm servers like Amazon’s A1 instances are often signficantly cheaper than x86 machines, users can achieve some immediate cost benefits by using this new system and running their containers on Arm.

For Docker, this partnership opens up new opportunities, especially in areas where Arm chips are already strong, including edge and IoT scenarios. Arm, similarly, is interested in strengthening its developer ecosystem by making it easier to develop for its platform. The easier it is to build apps for the platform, the more likely developers are to then run them on servers that feature chips from Arm’s partners.

“Arm’s perspective on the infrastructure really spans all the way from the endpoint, all the way through the edge to the cloud data center, because we are one of the few companies that have a presence all the way through that entire path,” Mohamed Awad, Arm’s VP of Marketing, Infrastructure Line of Business, said. “It’s that perspective that drove us to make sure that we engage Docker in a meaningful way and have a meaningful relationship with them. We are seeing compute and the infrastructure sort of transforming itself right now from the old model of centralized compute, general purpose architecture, to a more distributed and more heterogeneous compute system.”

Developers, however, Awad rightly noted, don’t want to have to deal with this complexity, yet they also increasingly need to ensure that their applications run on a wide variety of platform and that they can move them around as needed. “For us, this is about enabling developers and freeing them from lock-in on any particular area and allowing them to choose the right compute for the right job that is the most efficient for them,” Awad said.

Mesina noted that the promise of Docker has long been to remove the dependence of applications from the infrastructure they run on. Adding Arm support simply extends this promise to an additional platform. He also stressed that the work on this was driven by the company’s enterprise customers. These are the users who have already set up their systems for cloud-native development with Docker’s tools — at least for their x86 development. Those customers are now looking at developing for their edge devices, too, and that often means developing for Arm-based devices.

Awad and Messina both stressed that developers really don’t have to learn anything new to make this work. All of the usual Docker commands will just work.

 

Embrace raises $4.5M for its mobile application performance management platform

Embrace, an LA-based startup that offers a mobile-first application performance management platform, today announced that it has raised a $4.5 million funding round led by Pritzker Group Venture Capital. This brings the company’s total funding to $7 million. New investors Greycroft, Miramar Ventures and Vy Captial also participated in this round, as did previous investors Eniac Ventures, The Chernin Group, Techstars Ventures, Tikhon Bernstam of Parse and others.

Current Embrace customers include the likes of Home Depot, Headspace, OKCupid, Boxed, Thrive Market and TuneIn. These companies use the service to get a better view of how their apps perform on their users’ devices.

As Embrace CEO and co-founder Eric Futoran, who also co-founded entertainment company Scopely, argues, too many similar services mostly focus on crashes, yet those only constitute a small number of the actual user experience issues in most apps. “To a large extent, crashes are solved,” he told me. “The crash percentages are often 99.8 percent crash-free and yet users are still complaining.”

That’s because there are plenty of other issues beyond code exceptions, which many tools focus on almost exclusively, that can force an app to close (think memory issues or the OS shutting down the app because it uses too many CPU cycles). “To users, that looks like a crash. Your app closed. But in no way, that’s a crash from a technical perspective,” Futoran noted.

Raising this new round, Futoran told me, was pretty easy. Indeed, Pritzker approached the company. “It was not fundraising,” he said. “They sat us down and said, ‘we want to fund you guys,’ which I find pretty unusual. So I’ve been calling it a pre-emptive round.” He also noted that having Pritzker involved should help open up the mid-west market for Embrace, which is mostly focusing on enterprise customers (though Futoran’s definition of ‘enterprise’ includes the likes of digital-first companies like Headspace).

“We saw many organizations trust Embrace’s seamless and innovative optimization platform to quickly identify and resolve any user-impacting issues within their apps, and we’re optimistic about the future of the company in this growing market,” said Gabe Greenbaum, an LA-based Partner for Pritzker Group Venture Capital. “We look forward to this next stage in the company’s growth journey and are honored to partner with Eric and Fredric to help them achieve their vision.”

The company plans to use the new funding to increase its go-to-market capabilities, and grow its team to build out its technology.

 

 

VDOO secures $32M for a platform that uses AI to detect and fix vulnerabilities on IoT devices

Our universe of connected things is expanding by the day: the number of objects with embedded processors now exceeds the number of smartphones globally and is projected to reach some 18 billion devices by 2022. But just as that number is growing, so are the opportunities for malicious hackers to use these embedded devices to crack into networks, disrupting how these objects work and stealing information, a problem that analysts estimate will cost $18.3 billion to address by 2023. Now, an Israeli startup called VDOO has raised $32 million to address this, with a platform that identifies and fixes security vulnerabilities in IoT devices, and then tests to make sure that the fixes work.

The funding is being led by WRVI Capital and GGV Capital and also includes strategic investments from NTT DOCOMO (which works with VDOO), MS&AD Ventures (the venture arm of the global cyber insurance firm), and Avigdor Willenz (who founded both Galileo Technologies and Annapurna Labs, respectively acquired by Marvell and Amazon). 83North, Dell Technology Capital and David Strohm, who backed VDOO in its previous round of $13 million in January 2018, also participated, bringing the total raised by VDOO now to $45 million.

VDOO — a reference to the Hebrew word that sounds like “vee-doo” and means “making sure” — was cofounded by Netanel Davidi (co-CEO), Uri Alter (also co-CEO) and Asaf Karas (CTO). Davidi and Alter previously co-founded Cyvera, a pioneer in endpoint security that was acquired by Palo Alto Networks and became the basis for its own endpoint security product; Karas meanwhile has extensive experience coming to VDOO of working, among other places, for the Israeli Defense Forces.

In an interview, Davidi noted that the company was created out of one of the biggest shortfalls of IoT.

“Many embedded systems have a low threshold for security because they were not created with security in mind,” he said, noting that this is partly due to concerns of how typical security fixes might impact performance, and the fact that this has typically not been a core competency for hardware makers, but something that is considered after devices are in the market. At the same time, a lot of security solutions today in the IoT space have focused on monitoring, but not fixing, he added. “Most companies have good solutions for the visibility of their systems, and are able to identify vulnerabilities on the network, but are not sufficient at protecting devices themselves.”

The sheer number of devices on the market and their spread across a range of deployments from manufacturing and other industrial scenarios, through to in-home systems that can be vulnerable even when not connected to the internet, also makes for a complicated and uneven landscape.

VDOO’s approach was to conceive of a very lightweight implementation that sits on a small group of devices — “small” is relative here: the set was 16,000 objects — applying machine learning to “learn” how different security vulnerabilities might behave to discover adjacent hacks that hadn’t yet been identified.

“For any kind of vulnerability, using deep binary analysis capabilities, we try to understand the broader idea, to figure out how a similar vulnerability can emerge,” he said.

Part of the approach is to pare down security requirements and solutions to those pertinent to the device in question, and providing clear guidance to vendors for how to best avoid problems in the first place at the development stage. VDOO then also generates specific “tailor-made on-device micro-agents” to continue the detection and repair process. (Davidi likened it to a modern approach to some cancer care: preventive measures such as periodic monitoring checks; followed by a “tailored immunotherapy” based on prior analysis of DNA.)

It currently supports Linux- and Android-based operating systems, as well as FreeRTOS and support for more systems coming soon, Davidi said. It sells its services primarily to device makers, who can make over the air updates to their devices after they have been purchased and implemented to keep them up to date with the latest fixes. Typical devices currently secured with VDOO tech include safety and security devices such as surveillance cameras, NVRs & DVRs, fire alarm systems, access controls, routers, switches and access points, Davidi said.

It’s the focus on providing security services for hardware makers, in fact, that helps VDOO stand out from the others in the field.

“Among all startups for embedded systems, VDOO is the first to introduce a unique, holistic approach focusing on the device vendors which are the focal enabler in truly securing devices,” said Lip-Bu Tan, founding partner of WRVI Capital. “We are delighted to back VDOO’s technology, and the exceptional team that has created advanced tools to allow vendors to secure devices as much as possible without in-house security know-how, for the first time in many decades, I see a clear demand for security, as being raised constantly in many meetings with leading OEMs worldwide, as well as software giants.”

Over the last 18 months, as VDOO has continued to expand its own reach, it has picked up customers along the way after identifying vulnerabilities in their devices. Its dataset covers some 70 million embedded systems’ binaries and more than 16,000 versions of embedded systems, and it has worked with customers to identify and address 150 zero-day vulnerabilities and 100,000 security issues that would have potentially impacted 1.5 billion devices.

Interestingly, while VDOO is building its own IP, it is also working with a number of vendors to provide many of the fixes. Davidi says that VDOO and those vendors go through fairly rigorous screening processes before integrating, and the hope is that down the line there will more automation brought in for the “fixing” element using third-party solutions.

“VDOO brings a unique end-to-end security platform, answering the global connectivity trend and the emerging threats targeting embedded devices, to provide security as an essential enabler of extensive connected devices adoption. With its differentiated capabilities, VDOO has succeeded in acquiring global customers, including many top-tier brands. Moreover, VDOO’s ability to uncover and mitigate weaknesses created by external suppliers fits perfectly into our Supply Chain Security investment strategy,” said Glenn Solomon, managing partner at GGV Capital, in a statement. “This funding, together with the company’s great technology, skilled entrepreneurs and one of the best teams we have seen, will allow VDOO to maintain its leadership position in IoT security and expand geographies while continuing to develop its state-of-the-art technology.”

Valuation is currently not being disclosed.

Oracle turns to innovation hubs to drive cultural and business shift to cloud

Oracle was founded in 1977. While it’s not exactly IBM or GE, both of which date back to the late 19th and early 20th centuries respectively, it is old enough to be experiencing a fair bit of disruption in its own right. For a good part of its existence, it sold databases to some of the biggest companies in the world, but today as the market changes and shifts from on-prem data centers to the cloud, how does a company like Oracle make that transition?

Of course, Oracle has been making the shift to the cloud for the last several years, but it would be fair to say that it came late. Plus, it takes more than building some data centers and pushing out some products to change a company the size of Oracle. The company leadership recognizes this, and has been thinking at the highest levels of the organization about how to successfully transform into a cloud company from a cultural and business perspective.

To that end, Oracle has opened 5 innovation hubs over the last several years with locations in Austin, Texas; Reston, Virginia; Burlington, Massachusetts; Bangalore, India and Santa Monica, California. What are these centers hoping to achieve, and how will it extend the lessons learned to the rest of the company? Those are big questions Oracle must answer to make some headway in the cloud market.

Understanding the problem

Oracle seems to understand it has to do something different to change market perception and its flagging market position. Synergy Research, a firm that tracks cloud marketshare reports that the company is struggling

“For cloud infrastructure services (IaaS, PaaS, hosted private cloud services) — Oracle has a 2 percent share,” John Dinsdale, chief analyst and managing director at Synergy told TechCrunch. He added, “It is a top ten player but it is nowhere near the scale of the leading cloud providers; and its market share has been steadily eroding.”

The news is a bit better when it comes SaaS. “Along with SAP, Oracle is one of the leaders in the ERP segment. But enterprise SaaS is much broader than ERP and across all of enterprise SaaS it is the number 4 ranked provider behind Microsoft, Salesforce and Adobe. Oracle worldwide market share in Q4 was 6 percent,” Dinsdale said.

The company knows that it will take a vast shift to change from an organization that mostly sold software licenses and maintenance agreements. It pushed those hard, sometimes so hard that it left IT pros with a sour taste in their mouths. Today, with the cloud, the selling landscape has changed dramatically to a partnership model. The company knows that it must change too. The question is, how?

That will take an entirely new approach to product development, sales and marketing; and the innovation hubs have become a kind of laboratory where engineers can experiment with more focussed projects, and learn to present their ideas with goal of showing instead of telling customers what they can do.

And the young shall lead

One way to change the culture is to infuse it with fresh-thinking, smart young people and that’s what Oracle is attempting to do with these centers, where they are hiring youthful engineers, many right out of college, to lead the change with the help of more seasoned Oracle executives.

They are looking for ways to rethink Oracle’s cloud products, to pull the services together into packages of useful tools that helped solve a specific business problems from prescription opioid abuse to predicting avocado yields. The idea isn’t just to have a some section of the company where people work on dream projects. They want them to relate to real business problems that results eventually in actual sales and measurable results.

Hamza Jahangir, group vice president for the cloud solution hubs at Oracle says they look for people who want to dig into new solutions, but they want a practical streak in their innovation hub hires. “We don’t want just tinkerers. If the only problem you’re solving is that of your own boredom, that’s not the type of person we are looking for,” he said.

Oracle could be feeling cloud transition growing pains

Executive buy-in

The idea of the innovation center actually began with co-CEO Mark Hurd, according to Jahangir. He had been working for several years to change the nature of the sales force, the one that had a reputation of strong-arming IT pros, with a new generation by hiring people right out of college with a fresh approach.

Hurd didn’t want to stop with sales though. He began looking at taking that same idea of hiring younger employees to drive that cultural shift in engineering too. “About two years ago, Mark challenged us to think about how can we change the customer-facing tech workforce as the business model was moving to the cloud,” Jahangir said.

Hurd gave him some budget to open the first two centers in Austin and Reston and he began experimenting, trying to find the right kinds of employees and projects to work on. The funding came without of a lot of strings or conditions associated with it. Hurd wanted to see what could happen if they unleashed a new generation of workers and gave them a certain amount of freedom to work differently than the traditional way of working at Oracle.

Changing expectations

Jahangir was very frank when it came to assessing customer’s expectations around Oracle moving to the cloud. There has been a lot of skepticism and part of the reason for the innovation centers was to find practical solutions that could show customers that they actually had modern approaches to computing, given a chance.

The general customer stance has been, “We don’t believe you have anything real, and we need to see true value realized by us before we pay you any money,” he said. That took a fundamental shift to focussing on actual solutions. It started with the premise that the customers shouldn’t believe any of the marketing stuff. Instead it would show them.

Oracle’s Larry Ellison keeps poking AWS because he has no choice

“Don’t bother watching a Powerpoint presentation. Ask us to show you real solutions and use cases where we have solved real material problems — and then we can have a discussion.”

Even Chairman and company founder Larry Ellison recognizes the relationship and selling model needed to change as the company moves to the cloud. Jahangir relayed something he said in a recent internal meeting, “In the cloud we are now no longer selling giant monolithic software. Instead we are selling small bites of the apple. The relationship between the vendor and the buyer is becoming more like a consumer model.” That in turn requires a new way of selling and delivering solutions, precisely what they are trying to figure out at the innovation hubs.

Putting the idea to work

Once you have a new way of thinking, you have to put it to work, and as the company has created these various hubs, that has been the approach. As an example, one that isn’t necessarily original, but that puts Oracle features together in a practical way, is the connected patient. The patient wears a Fitbit-like monitor, uses a smart blood pressure cuff and a smart pill box.

The patient can then monitor his or her own health with these tools in a consolidated mobile application that pulls this data together for them using the Internet of Things cloud service, Oracle Mobile Cloud and Oracle Integration Cloud. What’s more, that information gets shared with the patient’s pharmacy and doctor, who can monitor the patient’s health and get warnings when there is a serious issue, such as dangerously high blood pressure.

Oracle’s database service offerings could be its last best hope for cloud success

Another project involved a partnership with Waypoint Robotics, where they demonstrated a robot that worked alongside human workers. The humans interacted with the robots, but the robot moved the goods from workstation to workstation acting as a quality control agent along the way. If it found defects or problems, it communicated that to the worker via a screen on the side of the unit, and to the cloud. Every interaction between the humans, goods and robot was updated in the Oracle cloud.

Waypoint Robotics Robot inspecting iPhones. Information on the display shows it communicating with the Oracle cloud. Photo: Ron Miller

One other project worked with farmers and distributors to help stores stay stocked with avocados, surely as good a Gen Z project as you are likely to find. The tool looks at weather data, historical sales and information coming from sensors at the farm, and it combines all of that data to make predictions about avocado yields, making use of Oracle Autonomous Data Warehouse, Oracle Analytics Cloud and other services from Oracle cloud stack.

Moving beyond the hubs

This type of innovation hub has become popular in recent years as a way to help stave off disruption, and Oracle’s approach is actually in line with this trend. While companies sometimes isolate them to protect them from negativity and naysayers in an organization, leaving them isolated often prevents the lessons learned from being applied to the broader organization at large, essentially defeating the very purpose of creating them in the first place.

Jahangir says that they are attempting to avoid that problem by meeting with others in the company and sharing their learnings and the kinds of metrics that they use in the innovation center to measure success, which might be different from the rest of the company.

He says to put Oracle on the customer agenda, they have to move the conversation from from religious battles, as he calls how people support or condemn tech from certain companies. “We have to overcome religious battles and perceptions. I don’t like to fight religion with more religion. We need to step out of that conversation. The best way we have seen for engaging developer community is to show them how to build really cool things, then we can hire developers to do that, and showcase that to the community to show that it’s not just lip service.”

The trick will be doing that, and perhaps the innovation centers will help. As of today, the company is not sharing its cloud revenue, so it’s hard to measure just how well this is helping contribute to the overall success of the company, but Oracle clearly has a lot of work to do to change the perception of the enterprise buyer about its cloud products and services, and to increase its share of the growing cloud pie. It hopes these innovations hubs will lead the way to doing that.

Jahangir recognizes that he has to constantly keep adjusting the approach. “The Hub model is still maturing. We are finding and solving new problems where we need new tooling and engagement models in the organization. We are still learning and evolving,” he said.

Digital Transformation Requires Total Organizational Commitment

Blueshift announces $15M Series B to expand AI-fueled cross-channel marketing tool

Blueshift is startup founded by tech industry veterans, who saw first-hand how difficult cross-channel marketing was. They decided to launch a company and build a cross-channel marketing platform from the ground up that uses AI and machine learning to make sense of the growing amount of customer data. Today, the startup announced a $15 million Series B round to keep it going.

The round was led by Softbank Ventures Asia, a fund focused on AI startups like Blueshift. Previous investors Storm Ventures and Nexus Venture Partners also participated. Today’s investment brings the total raised to $30 million, according the company.

Company co-founder and CEO Vijay Chittoor says the marketing landscape is changing, and he believes that requires a new approach to allow marketers to take advantage of the multiple channels where they could be engaging with customers from a single tool.

“If you thought about the world of customer engagement at Walmart or Groupon [or any other retailer] 10 years ago, it was primarily an email problem. Today, we as customers, we’re interacting with these brands on not just email, but also on mobile notifications, Facebook custom audiences and WeChat [and across multiple other channels],” he explained.

He says that this has created a lot more data, which it turns out is a double-edged sword for marketing pros. “I think on one end, it’s exciting for a marketer or a CMO to have more data and more channels. It gives them more ways to connect. But at the same time, it’s also more challenging because now you have to make sense of thousand times more data. And you have to use it intelligently on not just one channel like email, but you’re now trying to make sense of data across 15 different channels,” Chittoor said.

This a crowded field with big players like Adobe, Salesforce and Oracle, among others, offering similar cross-channel, AI-fueled solution. In addition startups are attracting huge chunks of money to attack this problem, including Klayvio pulling in $150 million a couple of weeks ago and Iterable, which landed $50 million last month.

He says his company’s differentiator is the AI piece, and it is this piece that the company’s lead investor in this round has been focusing on in its investments. The company plans to use this round to continue building out its marketing platform and show marketers how to communicate intelligently across channels wherever the consumer happens to be. Customers include LendingTree, Udacity and BBC.

Marketing tech vendors need to find right balance between digital and human interactions