During the last week, multiple full-time positions have been posted to UMBCworks. Check out these key positions today!

ARNIC- On Campus Interviews

Entry Level Python Developer

UMBCworks ID: 9263391

Application Deadline: 2/26/14

Symantec
Software Quality Assurance Engineer in Columbia, MD
UMBCworks ID: 9263695
Application Deadline: 3/7/14

During the last week, multiple internships and research positions have been posted to UMBCworks. Check out these key positions today! Many more can be found by logging into UMBCworks!

WaveStrike

Software Engineer Intern

UMBCworks ID: 9263654

Application Deadline: 3/6/14

Quality Training Systems

Quality Training Systems Intern and Co-Op Position.

UMBC Deadline
Submit the Free Application for Federal Student Aid (FAFSA) by February 14 for priority consideration of financial aid. Include our school code, 002105, and keep your confirmation page for your records.

State Deadline

Maryland - March 1, 2014 by midnight, Central Time.

Federal Deadline

Online applications must be submitted by midnight Central Time, June 30, 2015. Any corrections or updates must be submitted by midnight Central Time, September 19, 2015.

What's Next?

• Beginning in late March electronic notifications will be sent advising students to view their Financial Aid Packages on myUMBC.
• All required documents must be submitted and reviewed before an award notice will be generated. Failure to respond within the requested timeframe may reduce your student’s financial aid funding.
• The awards listed on the initial Award Notification are based on full-time enrollment. If your student plans to enroll for less than 12 credits (undergrad students), they must contact the Office of Financial Aid and Scholarships to discuss how part-time enrollment may affect their award.
• If you would like to speak to the financial aid office concerning your student’s financial aid please have your student submit the Authorization to Disclose Information form.

For additional questions or concerns feel free to contact the Financial Aid Office directly at 410-455-2387 or finaid@umbc.edu.

The Princeton Neuroscience Institute (PNI) is offering a summer internship program for highly motivated undergraduates, providing education and hands-on research experience in the field of neuroscience. Participants will closely collaborate with students and faculty at the PNI on original research projects, and thereby gain invaluable first-hand experience on what it is like to be a neuroscientist.

Summer interns will also participate in group educational activities, including weekly foundational overview lectures from PNI faculty, lab meetings with the researchers with whom they will work, journal club reading groups, plus special forums on topics such as career advice and graduate school application preparation.

Applicants must be current undergraduates, US citizens, permanent residents, or foreign undergraduates attending a US educational institution.

No stipend, housing expenses or other compensation will be provided to most interns during this summer research program.  This opportunity is offered strictly as an educational experience for highly motivated students interested in pursuing neuroscience in graduate school and beyond.

The winter season this year on the east coast has been a little chilly.  Long winter nights and cold days can impact one’s mood and energy level. Some may find themselves even struggling with a bit of cabin fever. Well fortunately for UMBC students there are plenty of great things happening on campus that will keep them engaged regardless of the weather outside. Here are few tips that can be offered to your son or daughter during times of cabin fever:

·         Attend a weekly movie – Each week students have an opportunity to view showings of a popular movie in Lecture Hall 1.

·         Attend an Athletic Events – Spring sports are in full swing. Check out the schedules here!

·         Become involved or stay actively involved in campus organizations. Click here for more info.

·         Attend a UMBC event, these can be anything from bingo, video game tournaments, open mic nights and more. Check out myUMBC events page for all the events happening on campus. 

In this article, we’ll go over the concepts and techniques required to build a command line tool using Node.js and PhantomJS. Building a command line tool enables you to automate a process that would otherwise take a lot longer.

Command line tools are built in a myriad of languages, but the one we’ll focus on is Node.js.

What We’ll Cover

• Secret sauce
• Installing Node.js and npm
• Process
• Automation
• PhantomJS
• Squirrel
• How it works
• The code
• Packaging
• Publishing
• Conclusion

Secret Sauce

For those short on time, I’ve condensed the core process into three steps. This is the secret sauce to convert your Node.js script into a fully functioning command line tool. But do stick around to see what else I have to show you.

1. In your package.json file, include the following settings:

• "preferGlobal": "true"
• "bin": { "name-of-command": "path-to-script.js" }

The rest of the process is just deciding what functionality to implement.

Installing Node.js And npm

To install Node.js, you have a few options:

• OS-specific installer for Windows, Mac or binary;
• Homebrew: brew install node;
• Nave;
• NVM.

Note that npm is installed as part of Node.js; there is no separate installation.

To test that Node.js and npm are installed correctly, run the following commands in your terminal:

• node --version
• npm --version

Process

Let’s consider a sample process: generating an Application Cache manifest file.

In case you are unfamiliar with AppCache, it enables you to take your application offline by specifying pages and resources to cache in the event that the user loses their Internet connection or tries to access your application later offline.

Typically, you would create an appcache.manifest file, where you would configure the offline settings.

We won’t go into much detail about AppCache itself because that would distract us from the purpose of this article. Nevertheless, below are the lines for a sample file:

    CACHE MANIFEST
    
    CACHE:
    foo.jpg
    index.html
    offline.html
    styles.css
    behaviours.js
    
    NETWORK:
    *
    
    FALLBACK:
    / /offline.html
    

As you can see, we’ve specified the following:

• a JPG image,
• two HTML files,
• a CSS file,
• a JavaScript file.

These are the resources that we want to cache in case the user goes offline.

We’ve also specified that all other items requested by the user should require a network to be accessed.

Finally, we’ve stated that any file that should be cached but isn’t yet should redirect the user to a file named offline.html.

Automation

Having to manually look up all of the images, style sheets, scripts and other pages linked from a Web page would be tedious. Thus, we’re trying to automate the process of generating an AppCache manifest file.

We could do this by writing some Node.js code along with some additional tools, but that wouldn’t be very easy (even for the person writing the script), because we would need to open the code and tell it which Web page to interrogate.

We also want other people to have the benefit of this tool, without their needing to download a folder full of code, change certain lines of code and run commands to run the scripts.

This is why a command line tool would help.

PhantomJS

First, we want to figure out how to solve this problem.

We’ll use a tool named PhantomJS, which is a headless (i.e. chromeless) browser.

Specifically, it’s a headless WebKit browser, which provides a JavaScript API that we can tap into and that lets us do things such as open Web pages and analyze their network requests. (It does many other things, but those are the two fundamental aspects we’re interested in.)

We can use a Node.js module to load PhantomJS and interact with its API. We can then convert our code into a command line tool with relative ease using Node.js’s package manager, npm, and a package.json file.

Squirrel

Luckily, I’ve already done the work for you. It’s an open-source project named Squirrel.

To install it, run the command npm install -g squirrel-js.

Once it’s installed, you can use it by running the command squirrel [url]. For example, squirrel bbc.co.uk/news.

This would generate (in the current directory) an appcache.manifest file populated with all relevant page resources.

How It Works

I started Squirrel by first writing the relevant Node.js and PhantomJS code to incorporate the functionality I was after.

Then, I added a script that bootstraps that code and allows me to take arguments that configure how the code runs.

I ended up with two scripts:

• squirrel.js
• appcache.js

The first script sets up the work:

• We specify the environment in which we want the script to execute (in this case, Node.js).
• Parse the arguments passed by the user.
• Read an internal (i.e. dummy) appcache.manifest file.
• Open a shell child process, call PhantomJS and pass it the script that we want it to execute (in this case, appcache.js) and the dummy manifest file.
• When the second script finishes its work (collating the Web page data), return to this first script and display some statistical information to the user and generate the manifest file.

The second script processes the Web page that the user has requested:

• We take in the dummy manifest file.
• Create listeners for the page resources that are requested.
• Set the viewport size.
• Open the Web page and store the resources.
• Get all links from the page (by executing JavaScript code directly in the Web page).
• Convert the contents of the manifest file and inject the resources found, and then return that as a JSON file.

The Code

Now that you understand what the code does, let’s review it. I’ll show the code in its entirely, and then we’ll go through it piecemeal.

squirrel.js

    #!/usr/bin/env node
    
    var userArguments = process.argv.slice(2); // Copies arguments list but removes first two options (script exec type & exec location)
    
    if (userArguments.length > 1) {
        throw new Error('Only one argument may be specified (the URL for which you want to generate the AppCache.)');
    }
    
    var fs               = require('fs');
    var shell            = require('child_process').execFile;
    var phantomjs        = require('phantomjs').path;
    var scriptToExecute  = __dirname + '/appcache.js';
    var manifest         = __dirname + '/../appcache.manifest';
    var url              = userArguments[0];
    var manifestContent;
    var data;
    
    fs.readFile(manifest, bootstrap);
    
    function bootstrap(err, contentAsBuffer) {
        if (err) throw err;
    
        manifestContent = contentAsBuffer.toString('utf8');
    
        shell(phantomjs, [scriptToExecute, url, manifestContent], function(err, stdout, stderr) {
            if (err) throw err;
    
            // Sometimes an error in the loaded page's JavaScript doesn't get picked up or thrown,
            // but the error comes in via stdout and causes JSON parsing to break
            try {
                data = JSON.parse(stdout);
            } catch(err) {
                log('Whoops! It seems there was an error? You\'ll find the stack trace below.');
                error(err);
            }
    
            displayStatistics();
            createManifestFile();
        });
    }
    
    function displayStatistics() {
        log(''); // Adds extra line of spacing when displaying the results
        log('Links: '      + data.links);
        log('Images: '     + data.images);
        log('CSS: '        + data.css);
        log('JavaScript: ' + data.javascript);
    }
    
    function createManifestFile() {
        fs.writeFile(process.cwd() + '/appcache.manifest', data.manifestContent, function(err) {
            if (err) throw err;
    
            log('\nManifest file created');
        });
    }
    
    function log(message) {
        process.stdout.write(message + '\n');
    }
    
    function error(err) {
        process.stderr.write(err);
    }
    

The first line, #!/usr/bin/env node, is critical to the script being used in the shell. We have to tell the shell what process should handle the script.

Next, we have to retrieve the arguments passed to the command. If we run squirrel bbc.co.uk/news, then process.argv would be an array containing the following:

• the script execution type (node);
• the script being executed (squirrel.js);
• any other arguments (in this instance, only one, bbc.co.uk/news).

Ignore the first two arguments, and store the user-specific arguments so that we can reference them later:

    var userArguments = process.argv.slice(2);
    

Our script only knows how to handle a single argument (which is the page URL to load). The following line isn’t really needed because we’ll ignore any more than one argument, but it’s useful for the code to have clear intent, so we’ll throw an error if more than one argument is passed.

    if (userArguments.length > 1) {
        throw new Error('Only one argument may be specified (the URL for which you want to generate the AppCache.)');
    }
    

Because we’re using PhantomJS, we’ll need to open up a shell and call the phantomjs command:

    var shell = require('child_process').execFile;
    

We’ll also need to reference the bin directory, where the PhantomJS executable is stored:

    var phantomjs = require('phantomjs').path;
    

Next, store a reference to the script that we want PhantomJS to execute, as well as the dummy manifest file.

    var scriptToExecute = __dirname + '/appcache.js';
    var manifest        = __dirname + '/../appcache.manifest';
    var url             = userArguments[0];
    

Because the PhantomJS script that we’ll be executing needs a reference to the dummy manifest file, we’ll asynchronously read the contents of the file and then pass it on to a bootstrap function:

    fs.readFile(manifest, bootstrap);
    

Our bootstrap function does exactly what you would expect: start our application (in this case, by opening the shell and calling PhantomJS). You’ll also notice that Node.js passes the contents of the manifest as a buffer, which we need to convert back into a string:

    function bootstrap(err, contentAsBuffer) {
        if (err) throw err;
    
        manifestContent = contentAsBuffer.toString('utf8');
    
        shell(phantomjs, [scriptToExecute, url, manifestContent], function(err, stdout, stderr) {
            // code...
        });
    }
    

At this point in the execution of the code, we are in the appcache.js file. Let’s move over there now.

appcache.js

The purpose of appcache.js is to get information from the user-requested page and pass it back to squirrel.js for processing.

Again, I’ll show the script in its entirety, and then we’ll break it down. (Don’t worry, we won’t go over each line — only the important parts.)

    var unique     = require('lodash.uniq');
    var system     = require('system');
    var fs         = require('fs');
    var page       = require('webpage').create();
    var args       = system.args;
    var manifest   = args[2];
    var css        = [];
    var images     = [];
    var javascript = [];
    var links;
    var url;
    var path;
    
    bootstrap();
    pageSetUp();
    openPage();
    
    function bootstrap() {
        if (urlProvided()) {
            url = cleanUrl(args[1]);
        } else {
            var error = new Error('Sorry, a valid URL could not be recognized');
                error.additional = 'Valid URL example: bbc.co.uk/news';
    
            throw error;
    
            phantom.exit();
        }
    
        if (bbcNews()) {
            // We want to serve the responsive code base.
            phantom.addCookie({
                'name'  : 'ckps_d',
                'value' : 'm',
                'domain': '.bbc.co.uk'
            });
        }
    }
    
    function pageSetUp() {
        page.onResourceRequested = function(request) {
            if (/\.(?:png|jpeg|jpg|gif)$/i.test(request.url)) {
                images.push(request.url);
            }
    
            if (/\.(?:js)$/i.test(request.url)) {
                javascript.push(request.url);
            }
    
            if (/\.(?:css)$/i.test(request.url)) {
                css.push(request.url);
            }
        };
    
        page.onError = function(msg, trace) {
            console.log('Error :', msg);
    
            trace.forEach(function(item) {
                console.log('Trace:  ', item.file, ':', item.line);
            });
        }
    
        page.viewportSize = { width: 1920, height: 800 };
    }
    
    function openPage() {
        page.open(url, function(status) {
            links      = unique(getLinks());
            images     = unique(images);
            css        = unique(css);
            javascript = unique(javascript);
    
            populateManifest();
    
            // Anything written to stdout is actually passed back to our Node script callback
            console.log(JSON.stringify({
                links           : links.length,
                images          : images.length,
                css             : css.length,
                javascript      : javascript.length,
                manifestContent : manifest
            }));
    
            phantom.exit();
        });
    }
    
    function urlProvided() {
        return args.length > 1 && /(?:www\.)?[a-z-z1-9]+\./i.test(args[1]);
    }
    
    function cleanUrl(providedUrl) {
        // If no http or https found at the start of the URL...
        if (/^(?!https?:\/\/)[\w\d]/i.test(providedUrl)) {
            return 'http://' + providedUrl + '/';
        }
    }
    
    function bbcNews(){
        if (/bbc.co.uk\/news/i.test(url)) {
            return true;
        }
    }
    
    function getLinks() {
        var results = page.evaluate(function() {
            return Array.prototype.slice.call(document.getElementsByTagName('a')).map(function(item) {
                return item.href;
            });
        });
    
        return results;
    }
    
    function writeVersion() {
        manifest = manifest.replace(/# Timestamp: \d+/i, '# Timestamp: ' + (new Date()).getTime());
    }
    
    function writeListContentFor(str, type) {
        manifest = manifest.replace(new RegExp('(# ' + str + ')\\n[\\s\\S]+?\\n\\n', 'igm'), function(match, cg) {
            return cg + '\n' + type.join('\n') + '\n\n';
        });
    }
    
    function populateManifest() {
        writeVersion();
    
        writeListContentFor('Images', images);
        writeListContentFor('Internal HTML documents', links);
        writeListContentFor('Style Sheets', css);
        writeListContentFor('JavaScript', javascript);
    }
    

We begin by using PhantomJS’ API to create a new Web page:

    var page = require('webpage').create();
    

Next, we’ll check that a URL was provided and, if so, clean it into the format required (for example, by giving it an http protocol). Otherwise, we’ll throw an error and stop PhantomJS:

    if (urlProvided()) {
        url = cleanUrl(args[1]);
    } else {
        var error = new Error('Sorry, a valid URL could not be recognized');
        error.additional = 'Valid URL example: bbc.co.uk/news';
    
        throw error;
        phantom.exit();
    }
    

We also put in a check to see whether the URL passed was for bbc.co.uk/news and, if so, use PhantomJS to set a cookie that enables the responsive version of the website to load (the purpose being merely to demonstrate some of PhantomJS’ useful APIs, such as addCookie):

    if (bbcNews()) {
        phantom.addCookie({
            'name'  : 'ckps_d',
            'value' : 'm',
            'domain': '.bbc.co.uk'
        });
    }
    

For PhantomJS to be able to analyze the network data (so that we can track the style sheets, JavaScript and images being requested by the page), we need to use special PhantomJS handlers to interpret the requests:

    page.onResourceRequested = function(request) {
        if (/\.(?:png|jpeg|jpg|gif)$/i.test(request.url)) {
            images.push(request.url);
        }
    
        if (/\.(?:js)$/i.test(request.url)) {
            javascript.push(request.url);
        }
    
        if (/\.(?:css)$/i.test(request.url)) {
            css.push(request.url);
        }
    };
    

We’ll also use another PhantomJS API feature that enables us to determine the size of the browser window:

    page.viewportSize = { width: 1920, height: 800 };
    

We then tell PhantomJS to open the specified Web page. Once the page is open (i.e. the load event has fired), a callback is executed:

    page.open(url, function(status) {
        // code...
    });
    

In the callback, we store the resources that were found, and we call a function that replaces the contents of our string (the dummy manifest) with a list of each set of resources:

    page.open(url, function(status) {
        links      = unique(getLinks());
        images     = unique(images);
        css        = unique(css);
        javascript = unique(javascript);
    
        populateManifest();
    
        // Remaining code...
    });
    

Finally, we create a data object to hold statistics about the resources being requested, convert it to a JSON string, and log it using the console API.

Once this is done, we tell PhantomJS to exit (otherwise the process would stall):

    page.open(url, function(status) {
        // Previous code...
    
        console.log(JSON.stringify({
            links           : links.length,
            images          : images.length,
            css             : css.length,
            javascript      : javascript.length,
            manifestContent : manifest
        }));
    
        phantom.exit();
    });
    

Reviewing the code above, you might wonder how we get the data back to our squirrel.js script? Take another look at the console.log. The code has an odd side effect, which is that any code logged by PhantomJS is passed back to our shell callback (originally executed in squirrel.js).

Let’s revisit our squirrel.js script now.

Back to squirrel.js

    shell(phantomjs, [scriptToExecute, url, manifestContent], function(err, stdout, stderr) {
        if (err) throw err;
    
        try {
            data = JSON.parse(stdout);
        } catch(err) {
            log('Whoops! It seems there was an error? You\'ll find the stack trace below.');
            error(err);
        }
    
        displayStatistics();
        createManifestFile();
    });
    

The callback function is run when the PhantomJS script finishes executing. It is passed any errors that may have occurred and, if there are, then we throw the error:

if (err) throw err;

The other arguments are the standard output and error arguments provided by the shell. In this case, the standard output would be our JSON string, which we console.log’ed from appcache.js. We parse the JSON string and convert it back into an object so that we can present the data to the user who has run the squirrel command.

As a side note, we wrap this conversion in a try/catch clause to protect against Web pages that cause a JavaScript error to occur (the error is picked up by stdout, not stderr, thus causing the JSON parsing to break):

    try {
        data = JSON.parse(stdout);
    } catch(err) {
        error(err);
    }
    

Once we have our data, we call displayStatistics, which uses stdout to write a message to the user’s terminal.

Lastly, we call createManifestFile, which creates an appcache.manifest file in the user’s current directory:

    fs.writeFile(process.cwd() + '/appcache.manifest', data.manifestContent, function(err) {
        if (err) throw err;
    
        log('\nManifest file created');
    });
    

Now that we understand how the script works in its entirety, let’s look at how to allow others to download and install our work.

Packaging

For other users to be able to install our module, we’ll need to publish it to a public repository. The place to do this is the npm registry.

To publish to npm, you’ll need a package.json file.

The purpose of package.json is to specify the dependencies of the project you’re working on. In this instance, it specifies the dependencies required by Squirrel to do its job.

Below is Squirrel’s package.json file:

    {
      "name": "squirrel-js",
      "version": "0.1.3",
      "description": "Node.js-based CLI tool, using PhantomJS to automatically generate an Application Cache manifest file for a specified URL",
      "main": "lib/squirrel",
      "scripts": {
        "test": "echo "Error: no test specified" && exit 1"
      },
      "engines": {
        "node": ">=0.10"
      },
      "repository": {
        "type": "git",
        "url": "git://github.com/Integralist/Squirrel.git"
      },
      "preferGlobal": "true",
      "bin": {
        "squirrel": "lib/squirrel.js"
      },
      "dependencies": {
        "phantomjs": "~1.9.2-6",
        "lodash.uniq": "~2.4.1"
      },
      "keywords": [
        "appcache",
        "phantomjs",
        "cli"
      ],
      "author": "Mark McDonnell  (http://www.integralist.co.uk/)",
      "license": "MIT",
      "bugs": {
        "url": "https://github.com/Integralist/Squirrel/issues"
      },
      "homepage": "https://github.com/Integralist/Squirrel"
    }
    

You can read up on all of the properties of package.json in the npm registry.

The properties to note are these:

• "preferGlobal": "true"
• "bin": { "squirrel": "lib/squirrel.js" }

The first property indicates when a user has installed a module that you would prefer to be installed globally. In this case, we want it to be installed globally because then the user will be able to run the command anywhere in their system.

The second property indicates where the command will find the code required to execute the command.

To test that your command works, you’ll need to run the npm link command, which in this case creates a symlink from the squirrel command to the squirrel.js file.

Publishing

To publish your code, first register for an npm account.

You’ll need to verify the account via the command line. To do this, run npm adduser, which will ask you to specify a user name and password.

Once you’ve verified the account, you can publish your module to the npm registry using npm publish.

It could take a few minutes for the module to become publicly accessible.

Be aware that if you update the code and try to run npm publish without updating the package.json file’s version property, then npm will return an error asking you to update the version number.

Conclusion

This is just one example of the sort of command line tools you can develop with Node.js’ many features.

The next time you find yourself performing a repetitive task, consider automating the process with a CLI tool.

(al)

© Mark McDonnell for Smashing Magazine, 2014.