Site Reliability Engineers: “solving the most interesting problems”

Posted by Chris Reid, Sydney Staffing team

I recently sat down with Ben Appleton, a Senior Staff Software Engineer, to talk about his recent move from Software Engineer (SWE) on the Maps team to Site Reliability Engineering (SRE). In the interview, Ben explains why he transitioned from a pure development role to a role in production, and how his work has changed:

Chris: Tell us about your path to Google.
Ben: Before I joined Google I didn’t consider myself a “software engineer”. I went to the University of Queensland and graduated with a Bachelor’s Degree in Electrical Engineering and Mathematics, before going on to complete a Ph.D. My field of research was image segmentation, extending graph cuts to continuous space for analyzing X-rays and MRIs. At a conference in France I met a friend of my Ph.D. advisor’s, and he raved about Google, commenting that they were one of the only companies that really understood technology. I’d already decided academia wasn’t for me, so I interviewed for a general Software Engineering role at Google. I enjoyed the interviews, met some really smart people, and learned about some interesting stuff they were working on. I joined the Maps team in Sydney in 2005 and spent the next 6 years working on the Maps API.

Chris: Tell us about some of the coolest work you did for Google Maps, and how you applied your research background.
Ben: My background in algorithms and computational geometry was really useful. We were basically making browsers do stuff they’re not designed to do, such as rendering millions of vectors or warping images, inventing techniques as we went. On the server-side we focused on content distribution, pushing tiles or vectors from Google servers down through caches to the user’s browser, optimizing for load and latency at every stage. On the client-side, we had to make the most of limited processors with new geometric algorithms and clever prefetching to hide network latency. It was really interesting work.

Chris: I understand you received company-wide recognition when you were managing the Maps API team. Tell us more about what that entailed.
Ben: In September 2008, when managing the Maps API, my team received an award that was recognized Google-wide, which is a big honor. My main contributions were latency optimizations, stability, enterprise support, and Street View integration. The award was in recognition of strong sustained growth of the Maps API, in relation to the number of sites using it, and total views per day. Currently the Google Maps API is serving more than 600,000 websites.

Chris: So what prompted the move to Site Reliability Engineering (SRE)?
Ben: In my experience, a lot of software engineers don’t understand what SREs do. I’d worked closely with SREs, particularly those in Sydney supporting Maps, and had formed a high opinion of them. They’re a very strong team - they’re smart and they get things done. After 6 years working on the Maps API I felt it was time for a change. In Sydney there are SWE teams covering most of the product areas, including Chrome and Apps, Social and Blogger, Infrastructure Networking and the Go programming language, as well as Maps and GeoCommerce. I talked to all of them, but chose SRE because in my opinion, they’re solving the most interesting problems.

Chris: How would you describe SRE?
Ben: It really depends on the individual. At one end are the Systems Administrator types, sustaining ridiculously large systems. But at the other end are the Software Engineers like me. As SREs get more experienced this distinction tends to be blurred. The best SREs think programmatically even if they don’t do the programming. For me, I don’t see a difference in my day-to-day role. When I was working on the Maps API I was the primary on-call one week in three, whereas in SRE the typical on-call roster is one week in six. When you’re primary on-call it just means you’re the go-to person for the team, responsible for when something breaks or pushing new code into production. I was spending 50% of my time doing coding and development work, and as an SRE this has increased to 80%.

Chris: Wow! So as an SRE in Production, you’re spending less time on-call and more time writing code than you were as a SWE on the Maps team?
Ben: Yes! I’m not managing a team now, but I’m definitely spending more time coding than I was before. I guess the average SRE spends 50% of their time doing development work, but as I said, it depends on the person and it ranges from 20-80%.

Chris: What does your team do?
Ben: In Sydney there are SRE teams supporting Maps, Blogger, App Engine, as well as various parts of the infrastructure and storage systems. I’m working on Blobstore, an infrastructure storage service based on Bigtable which simplifies building and deploying applications that store users' binary data (BLOBs, or "Binary Large OBjects"). Example BLOBs include images, videos, or email attachments - any data objects that are immutable and long-lived. The fact that we're storing user data means that Blobstore must be highly available for reads and writes, be extremely reliable (so that we never lose data), and be efficient in terms of storage usage (so that we can provide large amounts of storage to users at low cost).

Chris: Tell us more about some of the problems you’re solving, and how they differ with those you faced as a SWE in a development role.
Ben: With the massive expansion in online data storage, we’re solving problems at a scale never before seen. Due to the global nature of our infrastructure, we think in terms of load balancing at many levels: across regions, across data centers within a region, and across machines within a data center. The problems we’re facing in SRE are much closer to the metal. We’re constantly optimizing resource allocation and efficiency and scalability of Google’s massive computer systems, as opposed to developing new features for a product like Maps. So the nature of the work is very similar to SWE, but the problems are bigger and there is a strong focus on scale.

Chris: Are you planning on staying in SRE for a while?
Ben: Yeah. I signed up for a six month rotation program called “Mission Control,” the goal of which is to teach engineers to understand the challenges of building and operating a high reliability service at Google scale. In other words, it’s an SRE training program. In my first three months of Mission Control I’ve been on-call twice, and always during office hours so there were SREs to help me when I got into trouble...which I did. I’ve got no intention of going back to SWE at the end of the six months and plan to stay in SRE for at least a few years. Right now the problems seem more interesting. For example, last year’s storage solutions are facing additional strain from the growth of Gmail, Google+ and Google Drive. So you’re constantly reinventing.

Chris: What advice do you have for Software Engineers contemplating a role in SRE?
Ben: SRE gives you the opportunity to work on infrastructure at a really big scale in a way you don’t get to in SWE. Whereas SWE is more about developing new features, SRE is dealing with bigger problems and more complex engineering due to the sheer scale. SRE is a great way to learn how systems really work in order to become a great engineer.

If you’re interesting in applying for a Site Reliability Engineering role, please note that we advertise the roles in several different ways to reflect the diversity of the team. The two main roles are “Software Engineer, Google.com” and “Systems Engineer, Google.com”. We use the term “Google.com” to signify that the roles are in Production as opposed to R&D. You can find all the openings listed on the Google jobs site. We’re currently hiring across many regions, including Sydney in Australia, and of course Mountain View in California.

FxGqlC: DailyRollingFileAppender in log4j/log4net/log4cxx

Using log4j or one of the ports (like log4net or log4cxx), you can configure the appender to "roll" to a new file every day:
log4j.rootLogger=INFO, logfile

# logfile appender: writes its output to a file that is rolled each midnight.
log4j.appender.logfile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logfile.File=c:/logs/MyLogFile.log
log4j.appender.logfile.Append=true
log4j.appender.logfile.DatePattern='.'yyyy-MM-dd
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d{}{GMT} %X{pid} %X{pname} [%t-%X{tname}] %-5p - %c %m%n


This gives you one log file per day, suffixed with the date, except the last day:

• MyLogFile.log.2012-07-21
• MyLogFile.log.2012-07-22
• MyLogFile.log

When running the query "select distinct $filename from [*.*]" in FxGqlC, you get this list:

MyLogFile.log

MyLogFile.log.2012-07-21

MyLogFile.log.2012-07-22

So, the most recent file is scanned first, because this is the "ascending" order as returned by the operating system.

A workaround for this is to change the query like this: "select distinct $filename from [*.log.2*], [*.log]"

This workaround also applies to the regular RollingFileAppender, based on filesize instead of date.

A future feature will be to sort the files on modification date, by extending the FROM-clause option -fileorder.  Possibly in version 2.3.

Update: the FROM-clause option -fileorder has been extended in v2.3 to allow an order based on modification time.  For more information, have a look at:
https://sites.google.com/site/fxgqlc/home/fxgqlc-manual/changes-in-fxgqlc-2-3

Gource video of FxGqlC 2.2

I refreshed the Gource video of FxGqlC to reflect the latest version:

This time, the video was created on Ubuntu, and it is even simpler then on Windows.  These are the commands that I've run in a terminal window:

• sudo apt-get install gource
• sudo apt-get install ffmpeg
• cd ~/Projects/FxGqlC
  (the GIT-directory where the FxGqlC source code is located)
• gource -s 0.25 -title 'FxGqlC' -i 1000 -o /tmp/fxgqlc-gource.ppm
  (-s 0.25 to speed up to 4 days per second, -i 1000 to prevent fading out of idle items)
• avconv -y -r 25 -f image2pipe -vcodec ppm -i /tmp/fxgqlc-gource.ppm -vcodec wmv1 -r 25 -same_quant /tmp/fxgqlc-gource.wmv
• upload the video /tmp/fxgqlc-gource.wmv to youtube

Initially, I used ffmpeg as on Windows which worked without problems, but I got this warning:

*** THIS PROGRAM IS DEPRECATED ***

This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.

Migrating to "avconv" was no problem, since the exact same parameters could be used with it, except for the parameter -sameq which is replaced by -same_quant.

Clock problem when running Windows and Linux on same computer

When you install both Windows and Linux on the same machine (using dual/multi-boot), you get into an annoying problem with the system time:

• Linux uses the internal clock as UTC/GMT
• Windows uses the internal clock as Local Time.

On my box, in Belgium, this currently gives a difference of 2 hours (1 hour + 1 hour for daylight saving time) between my Windows 7 and Ubuntu.  After some time, the time is corrected because the operating systems syncs its time with an NTP-server on the internet.  But after booting into the other operating system, the problem repeats itself evidently.

This problem can be solved either in Windows or in Ubuntu.  I have chosen to change the Windows configuration:

• Start regedit.exe
• Go to the key:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation
• If it doesn't exist, add a new key (or change the existing):
  Set "RealTimeIsUniversal" as DWORD to "1"
• Reboot

You can also solve it in Linux:

• Open the file "/etc/default/rcS". When using a graphical shell like Unity or Gnome:
• start a terminal window by pressing Ctrl-Alt-T
• run "sudo gedit /etc/default/rcS" 
• Look for the line "UTC=yes", and change it to "UTC=no"

Creating symlinks and hardlinks in Explorer

A very useful tool to create  hardlinks, junctions and symbolic links ("symlinks") in Windows Explorer:
Link Shell Extension

It includes an explanation on hardlinks, junctions and symbolic links used in the NTFS file system:
http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html#hardlinks

Colorize GIT output

To colorize the output of GIT, you can enable the "color.ui" flag by executing:
git config --global --add color.ui true

More information can be found in the GIT book:
http://git-scm.com/book/ch7-1.html

FxGqlC 2.2 released

A new version of FxGqlC has been released.  There are many improvements, both in terms of performance and capabilities. 

Check it out on: https://sites.google.com/site/fxgqlc/home , and give it a try.

And many more things are in the pipeline, so come back in a few weeks for the next version.

The most important new features are:

• Change the working directory with the USE statement.  Similar to the cd/chdir commands in command prompts. USE [c:\temp]USE [../subdir]USE ['sub directory']

• Added support for variables.  Setting variables in select output (e.g. select )is not yet support
DECLARE @var string
SET @var = 'US' + ' ' + 'Open'
SELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on] WHERE [Tournament] = @var AND [Round] = 'The Final'
• System variable $filename: Returns current filename (without path).  The implementation of this system variable has been changed. Before v2.2, the full filename was returned (same behavior as current system variable $fullfilename).
  SELECT DISTINCT $filename FROM ['SampleFiles\*' -recurse]-- Returns:     AirportCodes.csv     AirportCodes.csv.zip     AirportCodesTwice.zip     CountryList.csv     IP2Country.csv.zip     Tennis-ATP-2011.csv
       AirportCodes2.csv     AirportCodes2.csv.zip

• System variable $fullfilename: Current full filename (with complete absolute path).
  This variable is only valid in the context of a running query.SELECT DISTINCT $fullfilename FROM ['SampleFiles\*' -recurse]-- Returns:     C:\Data\SampleFiles\AirportCodes.csv     C:\Data\SampleFiles\AirportCodes.csv.zip     C:\Data\SampleFiles\AirportCodesTwice.zip     C:\Data\SampleFiles\CountryList.csv     C:\Data\SampleFiles\IP2Country.csv.zip     C:\Data\SampleFiles\Tennis-ATP-2011.csv     C:\Data\SampleFiles\SubFolder\AirportCodes2.csv     C:\Data\SampleFiles\SubFolder\AirportCodes2.csv.zip

• Added FROM-clause options '-Heading=On', '-Heading=OnWithRule' and '-Heading=Off' (default).
  SELECT [Winner] from ['Tennis-ATP-2011.csv' -heading=on]     WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'-- Returns:     Djokovic N.

  
  

• Added possibility to show column headers in output, using !SET HEADING
  !SET HEADING OFFSELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on]     WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'-- Returns:     Djokovic N.
  
  !SET HEADING ONSELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on]      WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'-- Returns:     Winner     Djokovic N.
  !SET HEADING ONWITHRULESELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on]      WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'
  -- Returns:     Winner     ======     Djokovic N.

• The -Heading option can also be used in the INTO-clause:
  SELECT [Winner]      INTO ['US OPEN Winner.txt' -heading=onwithrule]     FROM ['Tennis-ATP-2011.csv' -heading=on]      WHERE [Tournament] = 'US OPEN' and [Round] = 'The Final'
• Added support for VIEWs:
  CREATE VIEW Tennis AS
       SELECT [Tournament], [Winner]
            FROM ['Tennis-ATP-2011.csv' -heading=on]
            WHERE [Round] = 'The final'
  SELECT * FROM Tennis
  DROP VIEW Tennis
• Added support for parameterized VIEWs:
  CREATE VIEW Tennis(@file string, @round string) AS
       SELECT [Tournament], [Winner]
            FROM [@file -heading=on]
            WHERE [Round] = @round
  SELECT * FROM Tennis('Tennis-ATP-2011.csv', 'The final')
  DROP VIEW Tennis
• Added support for count(*) as alternative to count(<expression>):
  SELECT count(*) FROM ['Tennis-ATP-2011.csv' -heading=on]
• Added support for count(distinct <expression>) to count unique values:
  SELECT count(distinct [Tournament]) FROM ['Tennis-ATP-2011.csv' -heading=on] 
• Block comments are now also supported.
  SELECT distinct [Tournament] /* block comment */ FROM ['Tennis-ATP-2011.csv' -heading=on] -- line comment
• Added support for option -columndelimiter in FROM-clause and INTO-clause.  Until now, the tab character "\t" was always used as delimiter, which is still the default. The string specified is unescaped using the RegularExpression syntax (e.g. \t becomes a tab character).SELECT [Date], [Winner]
       INTO ['output.txt' -columndelimiter=';']
       FROM ['Tennis-ATP-2011.csv' -heading=on]
       WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'
  -- Output.txt contains:
       12/09/2011;Djokovic N.
• HAVING-clause: Add a filter that is applied AFTER the GROUP BY aggregation.
  SELECT [Winner], count(*) FROM ['Tennis-ATP-2011.csv' -heading=on] GROUP BY [Winner] HAVING count(*) > 60
  
  
• Added support for "alias" in FROM-clause, which makes it possible to link subquery columns to outer query columns.
  SELECT [Date], [Tournament], [Winner],      (     SELECT count(*)           FROM ['Tennis-ATP-2011.csv' -heading=on] [inner]           WHERE [outer].[Winner] = [inner].[Winner]     )      FROM ['Tennis-ATP-2011.csv' -heading=on] [outer]      WHERE [Round] = 'The Final'

• A startup script file is automatically executed when FxGqlC.exe is started in command mode (-c, -command), in file mode (-gqlfile) or in prompt mode (-p, -prompt).  This can be useful to create regularly used views or variables, or to execute any comand such as USE or SET.  The startup script file path can be configured using the startup option -autoexec <filename>.  When the startup option -autoexec is not present, the default startup script file "autoexec.gql" is searched, first in the current directory and then in the directory where FxGqlC.exe is located.