Slicing and dicing data for interactive visualization

Posted by Benjamin Yolken, Google Public Data Product Manager

A year ago, we introduced the Google Public Data Explorer, a tool that allows users to interactively explore public-interest datasets from a variety of influential sources like the World Bank, IMF, Eurostat, and the US Census Bureau. Today, users can visualize over 300 metrics across 31 datasets, including everything from labor productivity (OECD) to Internet speed (Ookla) to gender balance in parliaments (UNECE) to government debt levels (IMF) to population density by municipality (Statistics Catalonia), with more data being added every week.

Last week, as part of the launch of our dataset upload interface, we released one of the key pieces of technology behind the product: the Dataset Publishing Language (DSPL). We created this format to address a key problem in the Public Data Explorer and other, similar tools, namely, that existing data formats don’t provide enough information to support easy yet powerful data exploration by non-technical users.

DSPL addresses this by adding an additional layer of metadata on top of the raw, tabular data in a dataset. This metadata, expressed in XML, describes the concepts in the dataset, for instance “country”, “gender”, “population”, and “unemployment”, giving descriptions, URLs, formatting properties, etc. for each. These concepts are then referenced in slices, which partition the former into dimensions (i.e., categories) and metrics (i.e., quantitative values) and link them with the underlying data tables (provided in CSV format). This structure, along with some additional metadata, is what allows us to provide rich, interactive dataset visualizations in the Public Data Explorer.

With the release of DSPL, we hope to accelerate the process of making the world’s datasets searchable, visualizable, and understandable, without requiring a PhD in statistics. We encourage you to read more about the format and try it yourself, both in the Public Data Explorer and in your own software. Stay tuned for more DSPL extensions and applications in the future!

Business at Warp Speed - Get Ready for the New Digital Revolution

[Cross-posted from Associated Industries of Massachusetts.]

All around us, things are accelerating. People, data, and information are all moving faster than ever before, accelerating at a pace that defies our understanding of the world and what is (or was thought to be) possible. 4G speeds, Google Instant, Tweets and check-ins, Facebook sign-ups, mobile device activations, video uploads – everything around us is growing and moving at exponential rates. Consider some of the statistics: in 2008, 300 million people were on social networks. That number is now over 900 million. In 2008 1.5 million people had visited Twitter. Last year over 190 million. In 2008 Google’s Android mobile platform had just launched. There are now over 300,000 Android device activations every day, and that number is growing. (Check out a really fun video showing the growth of Android here.)

And consider what has been happening in Middle Eastern countries of late. Via mobile phones and social networks, information is spreading and people are organizing at rates never before possible, allowing for rapid mobilization of ideas and massive social change.

In this world of constant connectivity, there is no doubt that fast is better than slow. From a consumer standpoint, you reap the benefits directly – connect to friends all over the world on Facebook, video chat with them on Skype, watch videos of the Grammy’s on YouTube, download Eminem’s latest to your iPod, search on Google for the shoes he wore, buy them on online…wait, sold out online?…check your local retailer’s inventory on Google Product Search, have your Android phone give you directions to your local mall, text your friends that you’ll meet them there, check in on Foursquare when you arrive… and on and on it goes, the flow of digital information accelerating at a rate that seems almost unimaginable.

So how do businesses keep up and take part in this digital evolution, nay, revolution? How do companies make use of this acceleration to reach consumers in meaningful ways? How do they change their own behaviors to keep up with the world around them?

Companies should look to get ahead of the consumer and these digital trends by focusing on 4 key characteristics: Google calls them the "4 Be’s"…

Be relevant – Mobile is how the world will connect to the Internet in the future. Mobile will be bigger than desktop in 5 years. Mobile searches grew 500% in the last two years. Mobile is what is relevant for today's and tomorrow's consumer. What is your company’s mobile strategy? Do you have one? Can it scale for the future?

Be found – Search is still the web’s killer app. With the proliferation of information, Search is how people find it and make use of it. Can people find you? Can they find your business and the products or services you offer?

Be engaging – We no longer live in a push advertising world. Consumers want and expect a dialog with companies. Social media like Facebook and Twitter make this possible. Online video has not only changed the face of entertainment but also the way companies can communicate. Companies and brands need to figure out how to be part of the conversation and how to effectively engage users across the entire spectrum of social media. What is your social media strategy? How are you effectively using online video to engage users?

Be accountable – the Internet has made real-time marketing a reality. You can constantly and continuously improve your digital efforts by using powerful analytical tools and the data they provide to make smart decisions. Things like Google Analytics, Insights for Search and others offer more data than ever before. How are you using it to make the best decisions for your company?

Success or failure in these four areas will determine your future. As “screen time” becomes mostly digital, as mobile becomes the norm for the masses, as consumers take more control of the dialog with companies and brands, and as everything becomes more real-time, businesses will need to evolve to keep pace…or run the risk of getting left behind.

Posted by: Seth van der Swaagh, National Industry Manager, AdWords

Where does my data live?

Posted by Daniel Ford, Senior Mathematician

Have you ever wondered what happens when you upload a photo to Picasa, or where all your Gmail or YouTube videos are stored? How it is that you can read or watch them from anywhere at any time?

If you stored your data on a single hard disk, like the one in your personal computer, then the disk would eventually fail and your data would be lost forever. If you want to protect your data from the possibility of such a failure, you can store copies across many different disks so that if any one fails then you just access the data from another.

However, once storage systems get large enough, anything and everything can and does go wrong. You have to plan not just for disk failures but for server, network, and entire datacenter failures. Add to this software bugs and maintenance operations and you have a whole lot more failures.

Using measurements from dozens of Google data centers, we found that almost-simultaneous failure of many servers in a data center has the greatest impact on availability. On the other hand, disk failures have relatively little impact because our systems are specifically designed to cope with these failures.

Once you have a model of failures, you can also look at the impact of various design choices. Where exactly should you place your data replicas? How fast do you need recover from losing a disk or server? What encoding scheme or number of replicas of the data is enough, given a desired level of availability? For example, we found that storing data across multiple data centers reduces data unavailability by many orders of magnitude compared to having the same number of replicas in a single data center. The added complexity and potential for slower recovery times is worth it to get better availability, or use less storage space, or even both at the same time.

As you can see, something as simple as storing your photos, mail, or videos becomes a lot more involved when you want to be sure it's always available.

In our paper, Availability in Globally Distributed Storage Systems, we characterize the availability of cloud storage systems, based on extensive monitoring of Google's main storage infrastructure, and the sources of failure which affect availability. We also present statistical models for reasoning about the impact of design choices such as data placement, recovery speed, and replication strategies, including replication across multiple data centers.

MagnetStreet: From real estate to bridal customers, lessons learned . . .

[Cross-posted from the Google Retail Advertising Blog]

In 2005, MagnetStreet, a custom magnet developer, was 80% reliant on its Real Estate customers. Although Real Estate customers kept MagnetStreet employees busy from August to November, the rest of the year was stagnant. In order to resolve this staffing and seasonal production slump, to acquire new customers and to expand its product portfolio, MagnetStreet looked to Google and its suite of free tools.

MagnetStreet realized they were not utilizing an SEO or SEM strategy. So they leveraged Google's Keyword Tool to learn more about terms such as “magnets” and “promotional magnets", and how they could expand their keyword selection to be in front of actively searching magnet customers.

David Baird, Vice President of Marketing at MagnetStreet, was surprised to find out that “save-the-date wedding magnets” stood out as keywords of rapidly growing interest. This new product keyword trend spurred an information-gathering project that started with understanding what a save-the-date magnet was, and ended with a solution regarding how to enter the wedding market.

MagnetStreet was also pleased to learn, via Google Trends and Insights for Search, that this new venture revealed an interest peak in January, with a more consistent level of interest throughout the year than the real estate market. These Google tools also offered geographic data that gave MagnetStreet the opportunity to intelligently market to the local areas that showed the most interest in their product suite.

MagnetStreet continues to use Insights for Search to help them expand into new markets, discover additional product offerings, and to inform design, staffing and media decisions. They have expanded into selling wedding invitations and programs in addition to various occasion invitations and announcements.

When looking back at his business’ success, Mr. Baird credits Google’s tools with helping him to make smarter, low risk decisions and giving him the opportunity as a small business owner to compete with Big Box players.

Click here for more information on this success story.

Posted by Keri Overman, The Google Retail Team

Going Google across the 50 States: Kentucky window manufacturer leaves desktop software behind

Editor's note: Over 3 million businesses have adopted Google Apps. Today we’ll hear from Steve Stepp, IT Manager of Sun Windows, a manufacturer of high quality windows and doors serving Owensboro, Kentucky and surrounding areas. To learn more about other organizations that have gone Google and share your story, visit our community map or test drive life in the cloud with the Go Google cloud calculator.

Sun Windows is a family-owned business that dates back to the 1930’s when V. E. Anderson, Sr. invented and built storm windows in his garage at night and sold them door-to-door during the day. Today, Sun Windows is run by his grandson, Frank Anderson, and offers an expansive product line of high quality, energy efficient windows and doors with a focus on customer service.

The window and door business is seasonal, following trends in new construction and peaking in summer. At the height, we have around 120 employees made up of about 80% production workers who manufacture the products, and 20% office staff and outside sales who use email and other office software regularly. Keeping everyone connected and communicating effectively is one of my main goals.

We originally used a local provider for web hosting and email and there was a lot of downtime when email just wouldn’t work. I’d get phone calls from individuals throughout the company and would have to contact our email provider about once a month. Adding to this, we received significantly more spam than good email. Sun Windows even got flagged as a spammer because all our emails went through the local provider. We’re a small company and everyone wears a lot of hats so these issues took up a lot of time I didn’t really have.

I used Gmail at home and had even set up Google Apps for my personal website so I knew about its robust spam-filtering and other great features. Given all the email problems we were having at work, I decided to switch the company to Google Apps and have never looked back. The amount of spam in our inboxes is almost nothing and having web-based email accessible from any Internet connection is a big plus for everyone. At the time of the switch, I hadn’t even considered the added benefits of other products like Google Calendar and Google Docs.

After setting up email, we quickly created shared calendars to keep various departments organized, track company events and schedule customer visits for the field service unit. Then we slowly started to use Google Docs. Most people in the company were familiar and comfortable with desktop office software but once they realized the power of collaborating and sharing documents online, almost everyone switched to Google Docs. Production line supervisors use a spreadsheet to track labor hours at the plant, and sales reps create and share customer presentations. We’ve also moved existing documents over to Google Docs which we use to store files of any type.

Now when new computers are purchased, I don’t renew our Microsoft® Office licences. The company saves money but even more importantly, I save time in administering licenses, installations, security patches, and training. Google Apps has been one of the smartest decisions I’ve made for Sun Windows and I continually look for new ways to take advantage of it to improve how we work.

Posted by Steve Stepp, IT Manager, Sun Windows

A Runtime Solution for Online Contention Detection and Response

Posted by Jason Mars, Software Engineering Intern

In our recent paper, Contention Aware Execution: Online Contention Detection and Response, we have made a big step forward in addressing an important and pressing problem in the field of Computer Science today. This work appears in the 2010 Proceedings of the International Symposium on Code Generation and Optimization (CGO) and was awarded the CGO 2010 Best Presentation Award at the conference.

One of the greatest challenges when using multicore processors arise when critical resources, such as the on-chip caches, are shared by multiple executing programs. If these programs simultaneously place heavy demands on shared resources, the may be forced to "take turns," and as a result, unpredictable and abrupt slowdowns may occur. This unexpected "cross-core interference" is especially problematic when considering the latency sensitive applications that are found in Google's datacenters, such as web-search. The commonly used solution is to dedicate separate machines to each application, however this leaves the processing capabilities of multicore processors underutilized. In our work, we present the Contention Aware Execution Runtime (CAER) environment that provides a lightweight runtime solution that minimizes cross-core interference, while maximizing utilization. CAER leverages the ubiquitous performance monitoring capabilities present in current state-of-the-art multicore processors to infer and respond to cross-core interference and requires no added hardware support. Our experiments show that when using our CAER system, we are able to increase the utilization of the multicore CPU by 58% on average. Meanwhile CAER brings the performance penally due to allowing co-location from 17% down to just 4% on average.

Tips and tricks for Travel advertisers

In 2010 people researched their travel options more frequently, for longer periods of time, and on various online channels. This is good news for Travel marketers because you now have more opportunities to reach customers -- and many of these channels can help you more effectively promote your businesses. With marketing tools like YouTube and display ads, Travel marketers can actually show images of their destinations. For example, Hawaii’s official travel site showcases amazing beaches and palm trees on their YouTube channel. Don’t these videos make you want to take a trip to Hawaii?

We saw travel bookings pick up in 2010 and we’re expecting this trend to continue into 2011. This is the year of reaching your customers at every phase of the research process -- and compelling them with visual imagery. We’ve included our recommendations for doing so below:

Seasonality: Spring break travelers don’t start researching their options in March. Our research shows that the average traveler spends almost two months researching travel options before booking. As such, we suggest using Insights for Search to see when certain destinations are on the rise -- you can plan your search ad campaigns based on this seasonality. For example, searches for ‘hotel cancun’ actually peak in January.

Search: Our research shows that 78% of travel transactions involve research on a search engine.* In fact, the average traveler searches eight times prior to booking. In addition to timing your search ads campaigns with proper search seasonality, we recommend taking advantage of new search formats. For example, if you are advertising a destination, you can create a free listing on Google Places and include detailed information and pictures.

Content pages: Rather than consulting travel agents, consumers are turning to online reviews, videos, and blogs. This means consumers are more engaged and involved in the research process -- it also means you have more opportunities to reach them as they research their options. The average hotel booker, for example, conducts 20 research sessions on multiple sites prior to transacting.* What’s more, ad inventory on these sites is generally less competitive than search: The median Travel advertiser’s cost per acquisition on the Google Display Network is 2% less than that on search.** You can even measure the impact of your display ads using Campaign Insights.

YouTube: 81% of all travelers who watch online videos do so on YouTube. And the number of travel-related online videos has increased dramatically since 2009. Video is a compelling way to showcase travel destinations -- If you are marketing a destination site or a hotel, we recommend shooting video footage (it doesn’t have to be high-budget or flashy) and uploading this content to a YouTube Brand Channel and promoting this channel on your website. From there, you can explore advertising options to drive additional traffic to your channel.

Mobile: Personal travelers more than doubled their usage of mobile devices for travel purposes in the past year.** Research also shows that among travel consumers, purchase intent is five times higher on mobile ads than on desktop ads.*** New mobile ad technologies present a fantastic opportunity to reach these customers when they’re ready to actually take the leap of booking their trip. In particular -- and for Travel advertisers who do not have mobile websites -- we recommend enabling the Click to Call option, which allows customers to call you directly from their mobile phones.

The possibilities are endless -- and if you reach a customer at various phases throughout the research process with compelling messages and images, chances are he or she will be more likely to book with you. Have fun this year!

*Data from a ComScore Behavioral study.

** Data from an Internal Google study.

*** Data from an InsightExpress study.

Posted by: Sarah Travis, Team Manager, AdWords Travel Team