More Google Cluster Data

Posted by John Wilkes, Principal Software Engineer

Google has a strong interest in promoting high quality systems research, and we believe that providing information about real-life workloads to the academic community can help.

In support of this we published a small (7-hour) sample of resource-usage information from a Google production cluster in 2010 (research blog on Google Cluster Data). Approximately a dozen researchers at UC Berkeley, CMU, Brown, NCSU, and elsewhere have made use of it.

Recently, we released a larger dataset. It covers a longer period of time (29 days) for a larger cell (about 11k machines) and includes significantly more information, including:

• the original resource requests, to permit scheduling experiments
• request constraints and machine attriibutes
• machine availability and failure events
• some of the reasons for task exits
• (obfuscated) job and job-submitter names, to help identify repeated or related jobs
• more types of usage information
• CPI (cycles per instruction) and memory traffic for some of the machines

Note that this trace primarily provides data about resource requests and usage. It contains no information about end users, their data, or access patterns to storage systems and other services.

More information can be found via this link, which will (after a short questionnaire) take you to a site that provides access instructions, a description of the data schema, and information about how the data was derived and its meaning.

We hope this data will facilitate a range of research in cluster management. Let us know if you find it useful, are willing to share tools that analyze it, or have suggestions for how to improve it.

Loading .net 4.0 assemblies in PowerShell 2.0

.Net assemblies can be load using the command "Add-Type - Path ./MyDotNetAssembly.dll". The last versions of Visual Studio build .net assemblies for .Net Framework version 4.0.
By default, PowerShell uses the .Net Framework version 2.0 run-time, and then you get the following error message in PowerShell:

Add-Type : Could not load file or assembly 'file:///C:\...\MyDotNetAssembly.dll' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.
At line:1 char:13
+     Add-Type <<<<  -Path .\MyDotNetAssembly.dll
    + CategoryInfo          : NotSpecified: (:) [Add-Type], BadImageFormatException
    + FullyQualifiedErrorId : System.BadImageFormatException,Microsoft.PowerShell.Commands.AddTypeCommand


To resolve this problem you can change the config file of PowerShell to let it load the .Net Framework version 4.0 instead of version 2.0.  This can be done using the PowerShell.exe.config file.
Locate the program PowerShell.exe which is normally "C:\windows\System32\WindowsPowerShell\v1.0", and create/update the file PowerShell.exe.config beside it, with this configuration:

<?xml version="1.0"?> 
<configuration> 
    <startup useLegacyV2RuntimeActivationPolicy="true"> 
        <supportedRuntime version="v4.0.30319"/> 
        <supportedRuntime version="v2.0.50727"/> 
    </startup> 
</configuration>


On 64-bit machines, the location of PowerShell is the same, but the 32-bit version of PowerShell is located under "C:\windows\SysWOW64\WindowsPowerShell\v1.0".  When you use the 32-bit version of PowerShell, you should create/change the config file there too.

The same trick can be applied to other software that hosts PowerShell, such as PowerShell_ISE.exe.

Google Apps highlights – 11/18/2011

Cross-posted from the Official Google Blog

This is part of a regular series of Google Apps updates that we post every couple of weeks. Look for the label “Google Apps highlights" and subscribe to the series. - Ed.

The last few weeks have brought a fresh new look in Gmail, more mobile access options and simpler meeting scheduling tools. Millions of organizations using Google Apps can now use Google+ on their business and university accounts, and we launched a couple Apps-related Google+ Pages ourselves.

Gmail’s new look
Back in July we previewed Gmail’s new look, and a couple weeks ago we started letting people switch to the new design with one click. The refreshed interface makes it easier to follow conversations and spot the sender with profile pictures for each message. The new look also supports dynamic screen densities, so Gmail displays properly whether you’re viewing on a large desktop monitor or a smaller mobile screen. We also added a selection of beautiful HD themes to the existing gallery. Finally, we made it easier to perform advanced email searches using a panel of powerful search options that reveals with a single click.



Gmail app for iOS devices
This month we introduced the Gmail app for the iPhone, iPad and iPod touch, complete with mobile alerts for new mail, a responsive touch screen interface and Gmail mainstays like fast search, conversation view and address auto-complete.

Suggested meeting times in Google Calendar
We’ve heard how frustrating it can be to spend 15 minutes finding a good time for people to convene for a 30 minute meeting, so we made it easier to find a good meeting time in Google Calendar. The suggested times feature automatically reviews the availability of meeting invitees, and proposes event times that work for the whole group.

Google+ for organizations using Google Apps
Businesses, schools and organizations with Google Apps can now use Google+. Employees and students can create profiles, +1 things they like on the web, share interesting content with their circles and have live multi-person video chats with classmates, colleagues and friends. Organizations can also create their Google+ Pages—an organization’s identity on Google+ for customers, students or fans. We’re using Google+ Pages ourselves, so take a look at the Gmail and Google Enterprise pages, and circle us if you’d like to stay in the loop.

24x7 telephone support and improved mobile device management
This week, we introduced a couple other new benefits for Google Apps customers. Organizations of all sizes around the world can now call our support hotline at any time for all core service issue. Also new this week, we improved our mobile device management capabilities with an interface for administrators to view and deny mobile devices connecting to Google Apps, granular mobile policy controls, and the ability to visualize mobile usage trends across the organization.

Who’s gone Google?
Organizations large and small continue to amass around Google Apps. We’re thrilled to welcome a whole host of new customers including the Trinity Mirror Group (Britain’s largest newspaper publisher), startups such as JobFlo and UserTesting, organizations including the U.S. Holocaust Memorial Museum, and colleges like the University of Michigan and UT Austin. Welcome to all!

Which database files are almost full?

The SQL Server query below can be used to get a list of all database files (data and transaction log), and their current size and the size that is currently in use.  The query uses the dynamic view (DMV) sys.databasefiles, which is present as from SQL Server version 2005.
The query is similar to DBCC SHOWFILESTATS, which also works on SQL Server version prior to 2005, but doesn't include the transaction log files.

-- Retrieve the occupation of all files of the current database

select file_name(file_id) FileName,

      fileproperty(file_name(file_id), 'SpaceUsed') UsedSize,

      size TotalSize,

      ceiling(fileproperty(file_name(file_id), 'SpaceUsed') * 100.0 / size) UsedPrc,

      physical_name FileNameOnDisk

from sys.database_files

--where type = 1 -- (for filtering TransactionLog)

Aero Peek Alt-Tab problem in Windows 7

In Windows 7, I had an annoying problem with Alt-Tab. After pressing Alt-Tab, the window containing the thumbnails of all applications disappeared due to "Aero Peek". In fact, the window containing the thumbnails disappears to the background because "Aero Peek" is showing an another Window on top of it. This problem was seen on two machines (a laptop and a desktop). On most machines it is working fine.

 I could not resolve the problem for now, but as a workaround I have turned off "Aero Peek":

1. Start Control Panel
2. In the search box, type "Performance Information and Tools"
3. In the left panel, click "Adjust visual effects"
   
4. On the first tab "Visual Effects", disable "Enable Aero Peak"
   
5. Click "Apply" or "OK"

Who is connected to my SQL Server database?

-- Who is connected to my SQL Server database?

-- (SQL Server 2005 and later)

select s.session_id, s.login_time, s.host_name, s.program_name, s.login_name

from sys.dm_exec_sessions s

inner joinsys.dm_exec_requests r onr.session_id =s.session_id

where r.database_id = db_id()

Get closer to your customers with Google+

(Cross-posted on the Inside AdWords blog.)

Editorial note: Many of you have already verified your Place page via Google Places for business. We encourage you to continue maintaining the information in your Google Places account to help ensure that potential customers can find accurate information about your business on Google Maps and local search. You may also want to supplement your Google Places account with a Google+ Page for your business, which provides you with additional ways to engage and directly interact with customers using Google+. Currently, however, Google Places accounts and Google+ Pages must be managed separately. 

Since the initial launch of Google+ just a few months ago, we’ve welcomed over 40 million people and introduced more than 100 new features. For all these people, one important part is still missing - your business.

We want to help you make the same kinds of lasting connections with your customers online the way you can in real life. That’s why we are introducing Google+ Your Business, a collection of tools and products to help you get closer to your customers. At the heart of this is Google+ Pages, your business’s identity on Google+.



Google+ Pages: Have real conversations with the right people

To get your business on Google+, you first need to create a Google+ Page. On your page, you can post updates and news about your business, engage in conversations with your customers, send tailored messages to specific groups of people, and see how many +1’s you have across the web. Here are a few features of Google+ Pages that we think will help you build relationships:

Google+ Pages are at the heart of Google+ for Business

Hangouts

Sometimes you might want to chat with your customers face-to-face. For example, if you run a bookstore, you may want to invite an author to talk about her latest novel, or if you market a cosmetics line, beauty specialists might want to hold how-to sessions for makeup tips. Hangouts make this easy, by letting you have high-quality video chats with nine customers, with a single click. You can use Hangouts to get product feedback, help solve problems or simply get to know people better, all in real time.

Hangouts let you meet your customers, face to face

Circles

Circles allow you to group followers of your Page into smaller audiences. You can then share specific messages with specific groups. For example, you could create a Circle containing people who are your most loyal customers and offer them a special discount.

+1 button and Google+ badge: Inspire current customers to recommend new ones

Google+ Pages also help you deliver your great content to users in a way that’s easy to share with their friends. We introduced the +1 button as a way for your customers to recommend your business across the web - on Google search, in your ads, on sites across the web and on mobile devices. Now, your customers can +1 your Page, recommending your actual brand -- not just an individual ad or a site -- directly to their friends.

To help your customers find your page and start sharing, we have two buttons you can add to your site by visiting our Google+ badge configuration tool:

The Google+ icon is a small icon that directly links to your Page.



In the coming days, we’re introducing the Google+ badge, which lets people add your page to their circles, without leaving your site, to start getting updates from your business via Google+.



Make your +1’s count, improving the performance of your ads

Coming soon, we will also make it possible to link your Page to your AdWords campaigns for your site, so that all your +1s -- from your Page, your website, ads and search results -- will get tallied together and appear as a single total.

Your +1’s will be shown with your brand wherever it appears, including search, ads, Google+ and your website

Consumers will be able to see all the recommendations your business has received, whether they are looking at an ad, a search result or your page, meaning your +1’s will reach not only the 40 million users of Google+, but all the people who come to Google every day. In the coming days. you'll be able to link your page to your AdWords campaigns by following the instructions in the AdWords Help Center.

Or, if you're using AdWords Express from your Google Places account, you can link your ads to your Google+ Page by simply selecting the option to direct viewers of your ad to your Google+ Page, and entering the Google+ Page URL in the appropriate field.

You can link your Google+ Page to your AdWords account with Social Extensions. Social Extensions are available in the ad extensions tab in your AdWords account.

Bringing Google+ to the rest of Google

Our ultimate vision for Google+ is to transform the overall Google experience-- weaving identity and sharing into all of our products. Beginning today, we’re rolling out a new experimental feature, Direct Connect -- an easy way for your customers to find your Google+ Page on Google search. For eligible pages, when someone searches for your business with the ‘+’ sign before it Direct Connect will send them directly to your page. For example, try searching for ‘+YouTube’ on Google. Users will also be prompted to automatically add Pages they find through Direct Connect to their circles. Direct Connect will not work for everyone.

Direct Connect suggestions start populating as you type on Google.com

Just the beginning

We wanted to help you get your business on Google+ as soon as possible, so we’re opening the field trial for Google+ Pages to everyone today. Creating a Google+ Page only takes a few minutes. To get started, you’ll need a personal Google+ profile. If you don’t have a Google account, it’s very quick and easy to join. And if you are looking for inspiration, check out some of the companies that are already starting to set up their Pages:

To learn more about how Google+ works for your business, check out the Google+ Your Business site. We’re just getting started and have many more features in the coming weeks and months. To keep up to date on the latest news and tips, add the Google+ Your Business page to your circles. If you have ideas on how we can improve Google+ for your business, we’d love to hear them.

Posted by Dennis Troper, Product Management Director, Google+ Pages

Automatic start of Empathy when logging into Ubuntu

Empathy is a chat application compatible with Facebook, Google Talk, ...  It is installed by default in the current releases of Ubuntu.  

It is currently impossible to configure Empathy to start it when you log on to Ubuntu. With an easy workaround, it can be started automatically:

• Start a terminal window (e.g. Ctrl-Alt-T)
• Run: "sudo cp /usr/share/applications/empathy.desktop /etc/xdg/autostart"

http://live.gnome.org/Empathy

Garmin & OpenStreetMap

The GPX files that are exported from Garmin Mapsource cannot be loaded into OpenStreetMap.org. You receive an error because there are not always timestamps available in the waypoints of the track. When you use the Active Track instead of aSaved Track of your GPS, the GPX file is correctly imported in OpenStreetMap.org.

You can use this tool to convert your GPX tracks (containing Saved tracks) to one that is compatible with OpenStreetMap.org:
FxGpxOpenStreetMapConverter
Requires .Net Framework 3.5.
The source code is also available:
FxGpxOpenStreetMapConverter
You can use Visual C# Express Edition (free) to compile the source code.



http://www.openstreetmap.org/
http://www8.garmin.com/support/download_details.jsp?id=209

Discovering Talented Musicians with Acoustic Analysis

Posted by Charles DuHadway, YouTube Slam Team, Google Research

In an earlier post we talked about the technology behind Instant Mix for Music Beta by Google. Instant Mix uses machine hearing to characterize music attributes such as its timbre, mood and tempo. Today we would like to talk about acoustic and visual analysis -- this time on YouTube. A fundamental part of YouTube's mission is to allow anyone anywhere to showcase their talents -- occasionally leading to life-changing success -- but many talented performers are never discovered. Part of the problem is the sheer volume of videos: forty eight hours of video are uploaded to YouTube every minute (that’s eight years of content every day). We wondered if we could use acoustic analysis and machine learning to pore over these videos and automatically identify talented musicians.

First we analyzed audio and visual features of videos being uploaded. We wanted to find “singing at home” videos -- often correlated with features such as ambient indoor lighting, head-and-shoulders view of a person singing in front of a fixed camera, few instruments and often a single dominant voice. Here’s a sample set of videos we found.



Then we estimated the quality of singing in each video. Our approach is based on acoustic analysis similar to that used by Instant Mix, coupled with a small set of singing quality annotations from human raters. Given these data we used machine learning to build a ranker that predicts if an average listener would like a performance.

While machines are useful for weeding through thousands of not-so-great videos to find potential stars, we know they alone can't pick the next great star. So we turn to YouTube users to help us identify the real hidden gems by playing a voting game called YouTube Slam. We're putting an equal amount of effort into the game itself -- how do people vote? What makes it fun? How do we know when we have a true hit? We're looking forward to your feedback to help us refine this process: give it a try*. You can also check out singer and voter leaderboards. Toggle “All time” to “Last week” to find emerging talent in fresh videos or all-time favorites.

Our “Music Slam” has only been running for a few weeks and we have already found some very talented musicians. Many of the videos have less than 100 views when we find them.

And while we're excited about what we've done with music, there's as much undiscovered potential in almost any subject you can think of. Try our other slams: cute, bizarre, comedy, and dance*. Enjoy!

Related work by Google Researchers:
“Video2Text: Learning to Annotate Video Content”, Hrishikesh Aradhye, George Toderici, Jay Yagnik, ICDM Workshop on Internet Multimedia Mining, 2009.

* Music and dance slams are currently available only in the US.