The scripting languages like Perl and Ruby are very well-known for expressing the data crunching task into one-liner. With their command line flags like -p, -e and default variable $_ with regular expression. The job just gets easier. I believe functional languages are also good at one-liner, since lambda-like structure are good in data transformation. Here is a few I would do with Haskell.

number each line of files

ghc -e "print . (zip [1..]) . lines =<< getContents"
ghc -e "print . (filter (/= '\n'))  =<< getContents"

count lines

ghc -e "print . length . lines =<< getContents"

print specific line

ghc -e "print . (filter (\x -> (fst x) == 1)) . (zip [1..]) . lines =<< getContents"

Regular expression is relatively painful in Haskell. In order to use it you have to add the modules you would like to load in evaluation mode first.

:m +Text.Regex

Then hit with POSIX regular expression syntax.

ghc -e 'print . (map (\s -> subRegex (mkRegex "^[[:space:]]+") s "")) . lines =<< getContents'

I would add more if I can think of others.

Because of reading libdnn source code, I started to survey about the existing technologies around DNN. Aside from caffe and Theano, one emerging framework as a keyword keep showing up in my reading is Torch 7. Unusual part of Torch 7 is that it is implemented with luajit, which in the era of R and python is a quite noticeable difference. I have read from 云風’s blog where luajit is pretty good, just I don’t have time to get my feet wet. And now it’s time.

I use an arbitrary benchmark game to test its performance. It’s not based on a solid analysis nor scientific proof, but I think it’s enough to get some clues of how performant it is. The test is to generate 1G of random bytes with dd. The program should read these in 512-byte chunk as key, and trying to detect if there exists duplicate in this pool of byte stream.

#!/bin/bash dd if=/dev/urandom of=large_random.txt bs=1024 count=0 seek=$[1000*1000]

Let’s take a look of results. The same algorithm was implemented in lua, python, c++, haskell and coffee-script. luajit is lightning fast, at least in this case. It only took around one second on my Macbook Air 2013 model. What follows is haskell, where I am pretty impressed that haskell’s hash map could be fine tuned to so much fast, and the code is borrowed from some stack overflow thread long time ago. And then it is C++, it took around four seances. I think it is because I did extra work in copying buffer’s content to string, and that took extra amount of time. Or either it is the unordered_map is not that well optimised. The next is python, where most of the dictionary operations are implemented in C, so I wouldn’t be amazed by this speed.

The nodejs’s speed is quite disappointing, I am not sure it is the third party library for streaming dragging it slow or it is nodejs’s inherent problem. It took around fifteen seconds to finish, and io.js runs a little bit faster than node. I probably did something wrong from this result, any correction are welcomed.

➜  uniq-blocks git:(master) ✗ time luajit lua/uniq-blocks.lua large_random.txt
luajit lua/uniq-blocks.lua large_random.txt  0.87s user 0.30s system 98% cpu 1.183 total
➜  uniq-blocks git:(master) ✗ time haskell/dist/build/uniq-blocks/uniq-blocks large_random.txt
haskell/dist/build/uniq-blocks/uniq-blocks large_random.txt  1.99s user 0.24s system 98% cpu 2.263 total
➜  uniq-blocks git:(master) ✗ time cpp/a.out large_random.txt
cpp/a.out large_random.txt  3.75s user 0.33s system 99% cpu 4.098 total
➜  uniq-blocks git:(master) ✗ time python python/uniq-blocks.py large_random.txt
python python/uniq-blocks.py large_random.txt  4.54s user 1.41s system 99% cpu 5.974 total
➜  uniq-blocks git:(master) ✗ time node js/uniq-blocks.js large_random.txt
reading large_random.txt …
node js/uniq-blocks.js large_random.txt  15.17s user 0.71s system 99% cpu 15.921 total
➜  uniq-blocks git:(master) ✗ time /opt/boxen/homebrew/Cellar/iojs/1.2.0/bin/iojs js/uniq-blocks.js large_random.txt
reading large_random.txt …
/opt/boxen/homebrew/Cellar/iojs/1.2.0/bin/iojs js/uniq-blocks.js   13.24s user 0.62s system 99% cpu 13.931 total

With these figures, I am quite impressed by the luajit. I would definitely invest more of my time on it.

The various codes are as follows:

import os
import os.path
import sys

def dedupeFile(dd, fp):
    fd = os.open(fp, os.O_RDONLY)
    for block in iter(lambda : os.read(fd, 512), ‘’):
        dd.setdefault(block, 0)
        dd[block] = dd[block] + 1
    os.close(fd)
    return dd

dd = {}
dedupeFile(dd, sys.argv[1])
function usage(program_name)
    print(“Usage: “ .. program_name .. “ filename”)
end

function dedupeFile(file)
    local f = io.open(file, “rb”)
    local t = {}

    while true do
        local block = f:read(512)

        if not block then break end

        if t[block] == nil then
            t[block] = 1
        else
            t[block] = t[block] + 1
        end
    end

    f:close()
end

if arg[1] == nil then
    usage(arg[0])
    do return end
else
    filename = arg[1]
    dedupeFile(filename)
end
fs = require ‘fs’
path = require ‘path’
lazy = require ‘lazy’
BlockStream = require ‘block-stream’

printLine = (line) -> process.stdout.write line + ‘\n’

usage = -> printLine (“Usage: “ + process.argv[1] + “ <filename>”)

if process.argv.length < 3
    usage()
    process.exit(0)
else
    filename = process.argv[2]
    printLine(“reading “ + filename + “ …”)

    d = {}

    dedupeFile = (chunk) ->
        if chunk of d
            d[chunk] += 1
        else
            d[chunk] = 1

    block = new BlockStream(512)
    rstream = fs.createReadStream(filename).pipe(block)
    block.on(‘data’, (chunk) -> dedupeFile(chunk))
    block.on(‘end’, -> printLine(‘Done’))

import           Control.Monad
import qualified Data.ByteString.Lazy as LB
import           Data.Foldable
import           Data.HashMap.Strict hiding (foldl’)
import           Data.Int
import qualified Data.List            as DL
import           System.Environment

type DummyDedupe = HashMap LB.ByteString Int64

toBlocks :: Int64 -> LB.ByteString -> [LB.ByteString]
toBlocks n bs | LB.null bs = []
              | otherwise = let (block, rest) = LB.splitAt n bs
                            in block : toBlocks n rest

dedupeBlocks :: [LB.ByteString] -> DummyDedupe -> DummyDedupe
dedupeBlocks = flip $ DL.foldl’ (\acc block -> insertWith (+) block 1 $! acc)

dedupeFile :: FilePath -> DummyDedupe -> IO DummyDedupe
dedupeFile fp dd = LB.readFile fp >>= return . (`dedupeBlocks` dd) . toBlocks 512

main :: IO ()
main = do
  dd <- getArgs >>= (`dedupeFile` empty) . head
  putStrLn . show . (*512) . size $ dd
  putStrLn . show . (*512) . foldl’ (+) 0 $ dd
#include <unordered_map>
#include <string>
#include <fstream>
#include <iostream>
#include <array>

void dedupleFile(std::ifstream &f)
{
    std::unordered_map<std::string, int> M;
    std::array<char, 512> buffer = { 0 };
    char *ptr = &buffer[0];

    while (f.read(ptr, 512)) {
        std::string key(ptr);
        if (M.count(key) > 0) {
            M[key] += 1;
        } else {
            M[key] = 1;
        }
    }
}

int main(int argc, char *argv[])
{
    if (argc < 2) {
        std::cerr << “Usage: “ << argv[0] << “ <filename>” << std::endl;
        return 1;
    }

    std::ifstream is(argv[1], std::ifstream::binary);
    if (is) {
        dedupleFile(is);
    } else {
        std::cerr << “reading failed” << std::endl;
    }

    return 0;
}