Measuring Ad Effectiveness Using Geo Experiments

Posted by Lizzy Van Alstine and Jon Vaver, Quantitative Analysis Team

Advertisers want to be able to measure the effectiveness of their advertising. Many methods have been used to address this need, but the most rigorous and trusted of these are randomized experiments, which involve randomly assigning experimental units to control and test conditions. At Google, we have found that randomized geo experiments are a powerful approach to measuring the effectiveness of advertising.

Many advertising platforms allow advertising to be targeted by geographical region. In these experiments, we first assign geographic regions to test or control conditions and employ AdWords’ geo-targeted advertising capabilities to increase or decrease the regional advertising spend accordingly. The use of randomized assignments guards against potential hidden test/control biases that could impact the measurements. Our approach also accounts for seasonal changes that impact the volume and cost of advertising across the length of the experiment.

In this paper, we describe the application of geo experiments for measuring the impact of advertising on consumer behavior (e.g. clicks, conversions, downloads, etc.). This description includes the results of a geo experiment that our research team ran for a Google advertiser.

New TalkBin improvements make it easier to connect with customers

Businesses interact with customers all the time, but it can be challenging to get honest and useful feedback that can help improve your business. TalkBin makes this easy by giving your customers a direct and convenient way to reach you via text message. No more clumsy comment cards and surveys; instead, customers can simply use their mobile phones to text you comments and feedback the same way they’d text a friend.

Today, we’re excited to announce three improvements that make it even easier to manage customer feedback.

1) Getting started tutorial

When you log in to TalkBin, you’ll be welcomed with a handy educational tutorial that explains all the most important features. This makes learning how to use TalkBin a cinch for you and your employees.

2) Updated dashboard

Based on your feedback, we updated the user interface with a fresh new design that helps you to manage all of your settings in one convenient place. For example, now your Account Settings are just one click away from Location and Billing Settings.

3) Custom sign builder

In-store signs are key to getting quality feedback from your customers. The new sign building feature help you make and print custom signs in just a few minutes.

To learn more about how TalkBin can help your business, check out these firsthand perspectives from Fraiche Yogurt and P. Terry’s Burger Stand.

Also, for the rest of the month, Google Small Business Blog readers can get TalkBin for just $5/month per location (discounted from the regular $25/month per location). Just enter promotion code BLOGSMB when you sign up at talkbin.com/signup/user/. This offer expires December 31, 2011.

Posted by Qasar Younis, Product Manager, TalkBin

ACM Fellows for 2011

Posted by Alfred Spector, Google Research

Cross-posted with the Official Google Blog

Congratulations to three Googlers elected ACM Fellows

It gives me great pleasure to share that the Association for Computing Machinery (ACM) has announced that three Googlers have been elected ACM Fellows in 2011. The ACM is the world’s largest educational and scientific computing society, and the Fellows Program celebrates the exceptional contributions of leaders in the computing field. This year the society has selected Amit Singhal, Peter S. Magnusson and Amin Vahdat for their outstanding work, which has provided fundamental knowledge to the field.

The recently-named Fellows join 14 prior Googler ACM Fellows and other professional society honorees in exemplifying our extraordinarily talented people. On behalf of Google, I congratulate our colleagues. They embody Google’s commitment to innovation with impact, and I hope that they’ll serve as inspiration to students as well as the broader community of computer scientists.

You can read more detailed summaries of their achievements below, including the official citations from the ACM.

Dr. Amit Singhal, Google Fellow

For contributions to search and information retrieval

Since 2000, Dr. Amit Singhal has been pioneering search as the technical lead for Google's core search algorithms. He is credited with most of the information retrieval design decisions in Google Search – a massive system that has responded to hundreds of billions of queries. More than anyone, Amit has a deep understanding of Google’s entire algorithmic system. He is responsible for prioritization and has overseen the development of numerous algorithmic signals and their progression over time. He is the clear thought and managerial leader who has led critically important initiatives at the company. Among many other things, Amit catalyzed Universal Search, which returns multi-modal results from all available corpora; he was the force behind Realtime Search, which returns results from dynamic corpora with low latency; and he championed Google Instant, which returns search results as the user types.

Prior to joining Google, Amit boasted a prolific publication record averaging 5 publications/year from 1996-9 while at AT&T Labs. Since that time, you could say Google Search has been one long, sustained publication demonstrating a constant advancement in the state of the art of information retrieval.


Peter S. Magnusson, Engineering Director

For contributions to full-system simulation

Peter has made a tremendous impact by driving full-system simulation. His approach was so advanced, it can be used in real world production of commercial CPUs and prototyping of system software. Starting in 1991, Peter began to challenge the notion that simulators could not be made fast enough to run large workloads, nor accurate enough to run commercial operating systems. His innovations in simulator design culminated in Simics, the first academic simulator that could boot and run commercial multiprocessor workloads. Simics saw huge academic success and has been used to run simulations for research presented in several hundred subsequent publications.

Peter founded Virtutech in 1998 to commercially develop Simics, and he ultimately forged and became the leader in a new market segment for software tools. With Peter at the helm, Virtutech pushed Simics beyond several performance barriers to make it the first simulator to exceed 1 billion instructions per second and the first simulator to model over 1,000 processors. Peter joined Google in 2010 to work with cloud computing.


Dr. Amin Vahdat, Principal Engineer

For contributions to data center scalability and management

Amin’s work made an impact at Google long before he arrived here. Amin is known for conducting research through bold, visionary projects that combine creativity with careful consideration of the engineering constraints needed to make them applicable in real world applications. Amin’s infrastructure ideas have underpinned the shift in the computing field from the pure client-server paradigm to a landscape in which major web services are hosted “in the cloud” across multiple data centers. In addition to pioneering “third-party cloud computing” through his work on WebOS and Rent-A-Server in the mid-90s, Amin has made important advancements in managing wide-area consistency between data centers, scalable modeling of data center applications, and building scalable data center networks.

Amin’s innovations have penetrated and broadly influenced the networking community within academia and industry, including Google, and his research has been recapitulated and expanded upon in a number of publications. Conferences that formerly did not even cover data centers now have multiple sessions covering variants of what Amin and his team have proposed. At Google, Amin continues to drive next-generation data center infrastructure focusing on Software Defined Networking and new opportunities from optical technologies. This is emblematic of Amin’s ability to build real systems, and perhaps more significantly, convince people of their value.

Our second round of Google Research Awards for 2011

Posted by Maggie Johnson, Director of Education & University Relations

We’ve just finished the review process for the latest round of the Google Research Awards, which provide funding to full-time faculty working on research in areas of mutual interest with Google. We are delighted to be funding 119 awards across 21 different focus areas for a total of $6 million. The subject areas that received the highest level of support this time were systems and infrastructure, human-computer interaction, social and mobile. In addition, 24% of the funding was awarded to universities outside the U.S.

One way in which we measure the impact of the research award program is through surveys of Principal Investigators (PIs) and their Google sponsors (a Googler with whom grantees can discuss research directions, provide progress updates, engage in knowledge transfer, etc.). Here are some highlights from our most recent survey, covering projects funded over the last two years:

• 433 papers were published as a result of a Google research award
• 126 projects made data sets or software publicly available
• 63 research talks were given by sponsored PIs at Google offices

An important aspect of the program is that it often gives early career academics a head start on their research agenda. Many new PIs commented on how a Google research award allowed them to explore their initial ideas and build a foundation for obtaining more significant funding from other sources. This type of seed funding is especially hard to get in the current economic environment.

The goal of the research award program is to initiate and sustain strong collaborations with our academic colleagues. The collaborations take many forms, from working on a project together, to co-writing a paper, to coming to Google to give a research talk. Whatever the form, the most important aspect is building strong relationships that last. Case in point, many of our focused awards (multi-year, unrestricted grants that include access to Google’s tools, technology and expertise) started as Google research awards.

Congratulations to the well-deserving recipients of this round’s awards, and if you are interested in applying for the next round (deadline is April 15), please visit our website for more information.

Pedal to find your dream home with Google Maps

(Cross-posted on the Lat Long Blog.)

The features available in Google Maps are equally functional and fun. You can tour distant cities with Street View or map out a trip with multiple destinations using Custom Maps. Not only is Google Maps a great tool for everyday personal use, but it’s can also be used as a practical business tool. Such is the case for Matt Kolb, owner of Pedal to Properties.

Matt is an avid cyclist and a realtor based in Boulder, Colorado. In 2006, Matt decided to blend his hobby and career by founding his own real estate agency called Pedal to Properties. His company is built on the idea that by touring homes via bicycle, one can get a better sense of the local community and determine if a particular property is the right fit.

When Matt meets with clients, he locates various properties on Google Maps and creates a biking route of the houses they’ll visit that day, using Bicycling Directions as a guide. Home buyers are encouraged to interact with the map, using Street View to check out a property and its surrounding neighborhood, and using nearby search to take a look at local schools and businesses. Through this process Matt is able to narrow down viable homes for a specific buyer, making the experience enjoyable and time-efficient.

If you have interesting stories about how you use biking directions, Street View, or other Google Maps features to enhance your business, comment on our Google+ Page with #mygmapstory

Posted by Dave Kim, Product Marketing Manager

2011 Google China Faculty Summit in Hangzhou

Posted by Aimin Zhu, University Relationship Manager, Google China

We just wrapped up a highly successful 2011 Google China Faculty Summit in Hangzhou, China. On November 17 and 18, Googlers from China and the U.S. gathered with more than 80 faculty members representing more than 45 universities and institutes, including Tsinghua University, Peking University and The Chinese Academy of Sciences. The two-day event revolved around the theme of “Communication, Exploration and Expansion,” with day one covering research and day two focusing on academic development.

The summit provided a unique setting for both sides to share the results of their research and exchange ideas. Speakers included:

• Maggie Johnson, director of education and university relations at Google, presenting on innovation in Google research and global university relations programs,
• Dr. Boon-Lock Yeo, head of engineering and research for Google China, providing an overview of innovation in China engineering and corporate social responsibility efforts and accomplishments, and
• Prof. Edward Chang, director of research for Google China, delivering a keynote on mobile information management and retrieval.

The discussions on November 17 focused on two tracks, mobile computing and natural language processing, while discussions on November 18 focused on curriculum development with a special focus on Android app development. The attendees also spent time discussing joint research and development between universities and industry.

This summit is part of a continuing to effort to collaborate with Chinese universities in order to support education in China. Click here for a list of the variety of education programs we have launched there in recent years. We look forward to expanding partnership opportunities in the future.

More Google Cluster Data

Posted by John Wilkes, Principal Software Engineer

Google has a strong interest in promoting high quality systems research, and we believe that providing information about real-life workloads to the academic community can help.

In support of this we published a small (7-hour) sample of resource-usage information from a Google production cluster in 2010 (research blog on Google Cluster Data). Approximately a dozen researchers at UC Berkeley, CMU, Brown, NCSU, and elsewhere have made use of it.

Recently, we released a larger dataset. It covers a longer period of time (29 days) for a larger cell (about 11k machines) and includes significantly more information, including:

• the original resource requests, to permit scheduling experiments
• request constraints and machine attriibutes
• machine availability and failure events
• some of the reasons for task exits
• (obfuscated) job and job-submitter names, to help identify repeated or related jobs
• more types of usage information
• CPI (cycles per instruction) and memory traffic for some of the machines

Note that this trace primarily provides data about resource requests and usage. It contains no information about end users, their data, or access patterns to storage systems and other services.

More information can be found via this link, which will (after a short questionnaire) take you to a site that provides access instructions, a description of the data schema, and information about how the data was derived and its meaning.

We hope this data will facilitate a range of research in cluster management. Let us know if you find it useful, are willing to share tools that analyze it, or have suggestions for how to improve it.