sábado, 21 de julho de 2012

Clock problem when running Windows and Linux on same computer

When you install both Windows and Linux on the same machine (using dual/multi-boot), you get into an annoying problem with the system time:

  • Linux uses the internal clock as UTC/GMT
  • Windows uses the internal clock as Local Time.
On my box, in Belgium, this currently gives a difference of 2 hours (1 hour + 1 hour for daylight saving time) between my Windows 7 and Ubuntu.  After some time, the time is corrected because the operating systems syncs its time with an NTP-server on the internet.  But after booting into the other operating system, the problem repeats itself evidently.

This problem can be solved either in Windows or in Ubuntu.  I have chosen to change the Windows configuration:
  • Start regedit.exe
  • Go to the key:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation
  • If it doesn't exist, add a new key (or change the existing):
    Set "RealTimeIsUniversal" as DWORD to "1"
  • Reboot
You can also solve it in Linux:
  • Open the file "/etc/default/rcS". When using a graphical shell like Unity or Gnome:
    • start a terminal window by pressing Ctrl-Alt-T
    • run "sudo gedit /etc/default/rcS" 
  • Look for the line "UTC=yes", and change it to "UTC=no"

sexta-feira, 20 de julho de 2012

Creating symlinks and hardlinks in Explorer

A very useful tool to create  hardlinks, junctions and symbolic links ("symlinks") in Windows Explorer:
Link Shell Extension

It includes an explanation on hardlinks, junctions and symbolic links used in the NTFS file system:
http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html#hardlinks

quinta-feira, 19 de julho de 2012

Colorize GIT output

To colorize the output of GIT, you can enable the "color.ui" flag by executing:
git config --global --add color.ui true


More information can be found in the GIT book:
http://git-scm.com/book/ch7-1.html

quarta-feira, 18 de julho de 2012

FxGqlC 2.2 released

A new version of FxGqlC has been released.  There are many improvements, both in terms of performance and capabilities. 
Check it out on: https://sites.google.com/site/fxgqlc/home , and give it a try.
And many more things are in the pipeline, so come back in a few weeks for the next version.
The most important new features are:
  • Change the working directory with the USE statement.  Similar to the cd/chdir commands in command prompts. USE [c:\temp]USE [../subdir]USE ['sub directory']
  • Added support for variables.  Setting variables in select output (e.g. select )is not yet support
  • DECLARE @var string
    SET @var = 'US' + ' ' + 'Open'
    SELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on] WHERE [Tournament] = @var AND [Round] = 'The Final'
  • System variable $filename: Returns current filename (without path).  The implementation of this system variable has been changed. Before v2.2, the full filename was returned (same behavior as current system variable $fullfilename).
    SELECT DISTINCT $filename FROM ['SampleFiles\*' -recurse]-- Returns:     AirportCodes.csv     AirportCodes.csv.zip     AirportCodesTwice.zip     CountryList.csv     IP2Country.csv.zip     Tennis-ATP-2011.csv
         AirportCodes2.csv
         AirportCodes2.csv.zip
  • System variable $fullfilename: Current full filename (with complete absolute path).
    This variable is only valid in the context of a running query.SELECT DISTINCT $fullfilename FROM ['SampleFiles\*' -recurse]-- Returns:     C:\Data\SampleFiles\AirportCodes.csv     C:\Data\SampleFiles\AirportCodes.csv.zip     C:\Data\SampleFiles\AirportCodesTwice.zip     C:\Data\SampleFiles\CountryList.csv     C:\Data\SampleFiles\IP2Country.csv.zip     C:\Data\SampleFiles\Tennis-ATP-2011.csv     C:\Data\SampleFiles\SubFolder\AirportCodes2.csv     C:\Data\SampleFiles\SubFolder\AirportCodes2.csv.zip
  • Added FROM-clause options '-Heading=On', '-Heading=OnWithRule' and '-Heading=Off' (default).
    SELECT [Winner] from ['Tennis-ATP-2011.csv' -heading=on]     WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'-- Returns:     Djokovic N.

  • Added possibility to show column headers in output, using !SET HEADING
    !SET HEADING OFFSELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on]     WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'-- Returns:     Djokovic N.

    !SET HEADING ON
    SELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on]      WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'-- Returns:     Winner     Djokovic N.
    !SET HEADING ONWITHRULE
    SELECT [Winner] FROM ['Tennis-ATP-2011.csv' -heading=on]      WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'
    -- Returns:     Winner     ======     Djokovic N.
  • The -Heading option can also be used in the INTO-clause:
    SELECT [Winner]      INTO ['US OPEN Winner.txt' -heading=onwithrule]     FROM ['Tennis-ATP-2011.csv' -heading=on]      WHERE [Tournament] = 'US OPEN' and [Round] = 'The Final'
  • Added support for VIEWs:
    CREATE VIEW Tennis AS
         SELECT [Tournament], [Winner]
              FROM ['Tennis-ATP-2011.csv' -heading=on]
              WHERE [Round] = 'The final'
    SELECT * FROM Tennis
    DROP VIEW Tennis
  • Added support for parameterized VIEWs:
    CREATE VIEW Tennis(@file string, @round string) AS
         SELECT [Tournament], [Winner]
              FROM [@file -heading=on]
              WHERE [Round] = @round
    SELECT * FROM Tennis('Tennis-ATP-2011.csv', 'The final')
    DROP VIEW Tennis
  • Added support for count(*) as alternative to count(<expression>):
    SELECT count(*) FROM ['Tennis-ATP-2011.csv' -heading=on]
  • Added support for count(distinct <expression>) to count unique values:
    SELECT count(distinct [Tournament]) FROM ['Tennis-ATP-2011.csv' -heading=on] 
  • Block comments are now also supported.
    SELECT distinct [Tournament] /* block comment */ FROM ['Tennis-ATP-2011.csv' -heading=on] -- line comment
  • Added support for option -columndelimiter in FROM-clause and INTO-clause.  Until now, the tab character "\t" was always used as delimiter, which is still the default. The string specified is unescaped using the RegularExpression syntax (e.g. \t becomes a tab character).SELECT [Date], [Winner]
         INTO ['output.txt' -columndelimiter=';']
         FROM ['Tennis-ATP-2011.csv' -heading=on]
         WHERE [Tournament] = 'US OPEN' AND [Round] = 'The Final'
    -- Output.txt contains:
         12/09/2011;Djokovic N.
  • HAVING-clause: Add a filter that is applied AFTER the GROUP BY aggregation.
    SELECT [Winner], count(*) FROM ['Tennis-ATP-2011.csv' -heading=on] GROUP BY [Winner] HAVING count(*) > 60

  • Added support for "alias" in FROM-clause, which makes it possible to link subquery columns to outer query columns.
    SELECT [Date], [Tournament], [Winner],      (     SELECT count(*)           FROM ['Tennis-ATP-2011.csv' -heading=on] [inner]           WHERE [outer].[Winner] = [inner].[Winner]     )      FROM ['Tennis-ATP-2011.csv' -heading=on] [outer]      WHERE [Round] = 'The Final'
  • A startup script file is automatically executed when FxGqlC.exe is started in command mode (-c, -command), in file mode (-gqlfile) or in prompt mode (-p, -prompt).  This can be useful to create regularly used views or variables, or to execute any comand such as USE or SET.  The startup script file path can be configured using the startup option -autoexec <filename>.  When the startup option -autoexec is not present, the default startup script file "autoexec.gql" is searched, first in the current directory and then in the directory where FxGqlC.exe is located. 

terça-feira, 17 de julho de 2012

Reducing PDF file size

Using GhostScript, it is very easy to make PDF files smaller. Run this command:
gs -sDEVICE=pdfwrite -dPDFSETTINGS=/screen -dCompatibilityLevel=1.4 -dNOPAUSE -dQUIET -dBATCH -sOutputFile=NewFile.pdf OriginalFile.pdf

segunda-feira, 16 de julho de 2012

Subversion working copy locked

When you get the SubVersion (svn) error "Working copy <directory> locked" in Tortoise, you can try this to unlock the directory:

  • Open a command prompt, and change the directory to your locked subversion directory.
  • Run svn cleanup

sexta-feira, 13 de julho de 2012

Google at SIGMOD/PODS 2012



Over the years, SIGMOD has expanded beyond a traditional "database" conference to include several areas related to information management. This year’s ACM SIGMOD/PODS conference (on Management of Data, and Principles of Database Systems), held in Scottsdale, Arizona was no different. We were impressed by the wide variety of researchers from industry and academia alike the conference attracted, and enjoyed learning how others are pushing the limits of scalability in data storage and processing. In addition to an excellent set of papers on a large number of topics, we saw a couple of recurring themes:

1) Data Visualization
  • Pat Hanrahan from Stanford gave a keynote on some of the challenges involved in building systems to enable "data enthusiasts" to manage and visualize data. 

2) Big Data


As has been the case for the last couple of years, “Big Data" has been of ever-growing interest to the entire community, particularly from industry. Google presented a talk on F1, a new distributed database system we’ve built to power the AdWords system. A complex business application like AdWords has different requirements than many systems at Google that often use storage systems like Bigtable. We have a single database shared by hundreds of developers and systems, so we need the robustness and ease of use we’re used to from traditional databases. F1 is built to scale like Bigtable, without giving up the database features we also need, like strong consistency, ACID transactions, schema enforcement, and most importantly, SQL query.

There’s been a widespread trend over the last several years away from databases, towards highly scalable “NoSQL” systems. We don’t think that trade-off is necessary, and were happy to see several other speakers advocate a similar theme -- yes, databases are useful, and developers shouldn’t need to give up database features and ease of use in the name of scalability.

This theme was supported by an industry session on Big Data featuring talks from other companies: Facebook (TAO: How Facebook Serves the Social Graph), Twitter (Large-Scale Machine Learning at Twitter), and Microsoft (Recurring Job Optimization in Scope). Googler Kirsten LeFevre was a panelist on the "Perspectives on Big Data" panel organized by Surajit Chaudhuri from Microsoft, and also featuring Donald Kossmann from ETHZ, Sam Madden from MIT, and Anand Rajaraman from Walmart Labs. Last but not the least, Surajit Chaudhuri also gave an excellent keynote outlining some of the research challenges that the new era of "Big Data and Cloud" poses.

As has been the practice for several years now, to continue generating great interest in data management research, SIGMOD has been organizing panels such as this year's "New Research Symposium" (which included Anish Das Sarma from Google as a panelist).

In addition to sponsoring the conference, many Googlers attended contributing to a robust presence and affording us the opportunity to interact with the broader information management community. We've been pushing the frontiers of science with cutting-edge research in many aspects of data management, and we were eager to share our innovations and see what others have been working on. We found Amin Vahdat's keynote on the intersection of Networking and Databases to be a highlight of Google’s participation, which also included presenting papers, participating on panels, and taking part in planning and program committees:

Program Committee Members


Anish Das Sarma, Venkatesh Ganti, Zoltan Gyongyi, Alon Halevy (Tutorials Chair), Kristen LeFevre, Cong Yu

Talks


Symbiosis in Scale Out Networking and Data Management
Amin Vahdat, Google (Keynote)

F1-The Fault-Tolerant Distributed RDBMS Supporting Google's Ad Business
Jeff Shute, Mircea Oancea, Stephan Ellner, Ben Handy, Eric Rollins, Bart Samwel, Radek Vingralek, Chad Whipkey, Xin Chen, Beat Jegerlehner, Kyle Littlefield, Phoenix Tong (Googlers)

Finding Related Tables
Anish Das Sarma, Lujun Fang, Nitin Gupta, Alon Halevy, Hongrae Lee, Fei Wu, Reynold Xin, Cong Yu (Googlers)

Papers


CloudRAMSort: Fast and Efficient Large-Scale Distributed RAM Sort on Shared-Nothing Cluster
Changkyu Kim, Jongsoo Park, Nadathur Satish, Hongrae Lee (Google), Pradeep Dubey, Jatin Chhugani

Efficient Spatial Sampling of Large Geographical Tables
Anish Das Sarma, Hongrae Lee, Hector Gonzalez, Jayant Madhavan, Alon Halevy (Googlers)

Panels


Perspectives on Big Data Plenary Session: Privacy and Big Data 
Kristen LeFevre, Google

SIGMOD New Researcher Symposium - How to be a good advisor/advisee? 
Anish Das Sarma, Google

Overall, this year’s SIGMOD was a great conference, widely attended by researchers from industry and academia, and comprised of a very interesting mix of research presentations and discussions. Google had a good showing at the conference, and we look forward to continuing this trend in the coming years.