Category Archives: Tech

Excel “Pull” Function: Creating dynamic links to closed workbooks

I was working on a project this week that required aggregation of data from many, many different Excel workbooks (and various tabs within those workbooks) into one unified dashboard. Now, this can be accomplished with traditional linking, but anyone who has tried to link more than a few workbooks together will tell you this is a pain to maintain and often not very reliable.

One useful function for creating dynamic links within the same workbook is INDIRECT. Instead of hardcoding cell and worksheet references into a formula, INDIRECT allows pieces of a formula to be dynamically updated using variables. Sounds pretty good right? Well, INDIRECT has an Achilles’ heel. It only works if you are linking within the same workbook.

Need dynamic links to a closed workbook? Bill Gates says, “You’re out of luck.”

Fortunately, after searching around, I found an old Usenet post from 2004 by a fellow named Harlan Grove. He wrote an Excel function in VB called PULL which essentially fixes INDIRECT’s fatal flaw and allows the user to create links to closed workbooks. Just copy the code below into a Module within the workbook of your choice. Then you can use the PULL function just like any other.

Example:
A1 =C:\work\
B1 = data\
C1 = [ClosedBook.xls]Sheet1′!$A1
E1 = PULL(“‘”&A1&B1&C1)

'----- begin VBA -----
Function pull(xref As String) As Variant
'inspired by Bob Phillips and Laurent Longre
'but written by Harlan Grove
'-----------------------------------------------------------------
'Copyright (c) 2003 Harlan Grove.
'
'This code is free software; you can redistribute it and/or modify
'it under the terms of the GNU General Public License as published
'by the Free Software Foundation; either version 2 of the License,
'or (at your option) any later version.
'-----------------------------------------------------------------
'2004-05-30
'still more fixes, this time to address apparent differences between
'XL8/97 and later versions. Specifically, fixed the InStrRev call,
'which is fubar in later versions and was using my own hacked version
'under XL8/97 which was using the wrong argument syntax. Also either
'XL8/97 didn't choke on CStr(pull) called when pull referred to an
'array while later versions do, or I never tested the 2004-03-25 fix
'against multiple cell references.
'-----------------------------------------------------------------

'2004-05-28
'fixed the previous fix - replaced all instances of 'expr' with 'xref'
'also now checking for initial single quote in xref, and if found
'advancing past it to get the full pathname [dumb, really dumb!]
'-----------------------------------------------------------------
'2004-03-25
'revised to check if filename in xref exists - if it does, proceed;
'otherwise, return a #REF! error immediately - this avoids Excel
'displaying dialogs when the referenced file doesn't exist
'-----------------------------------------------------------------
Dim xlapp As Object, xlwb As Workbook
Dim b As String, r As Range, C As Range, n As Long
'** begin 2004-05-30 changes **

'** begin 2004-05-28 changes **
'** begin 2004-03-25 changes **
n = InStrRev(xref, "\")
If n > 0 Then
If Mid(xref, n, 2) = "\[" Then
b = Left(xref, n)
n = InStr(n + 2, xref, "]") - n - 2
If n > 0 Then b = b & Mid(xref, Len(b) + 2, n)
Else
n = InStrRev(Len(xref), xref, "!")
If n > 0 Then b = Left(xref, n - 1)
End If

'** key 2004-05-28 addition **
If Left(b, 1) = "'" Then b = Mid(b, 2)
On Error Resume Next
If n > 0 Then If Dir(b) = "" Then n = 0
Err.Clear
On Error GoTo 0
End If

If n <= 0 Then
pull = CVErr(xlErrRef)
Exit Function
End If
'** end 2004-03-25 changes **
'** end 2004-05-28 changes **
pull = Evaluate(xref)

'** key 2004-05-30 addition **
If IsArray(pull) Then Exit Function
'** end 2004-05-30 changes **

If CStr(pull) = CStr(CVErr(xlErrRef)) Then
On Error GoTo CleanUp 'immediate clean-up at this point

Set xlapp = CreateObject("Excel.Application")
Set xlwb = xlapp.Workbooks.Add 'needed by .ExecuteExcel4Macro

On Error Resume Next 'now clean-up can wait

n = InStr(InStr(1, xref, "]") + 1, xref, "!")
b = Mid(xref, 1, n)

Set r = xlwb.Sheets(1).Range(Mid(xref, n + 1))

If r Is Nothing Then
pull = xlapp.ExecuteExcel4Macro(xref)

Else
For Each C In r
C.Value = xlapp.ExecuteExcel4Macro(b & C.Address(1, 1, xlR1C1))
Next C

pull = r.Value

End If

CleanUp:
If Not xlwb Is Nothing Then xlwb.Close 0
If Not xlapp Is Nothing Then xlapp.Quit
Set xlapp = Nothing

End If

End Function
'----- end VBA -----

R for SAS and SPSS Users, A Review

If you’ve followed me for any length of time, you’ll know that I’m bullish on good data analysis. I am not a programmer by training but, through a variety of projects, I’ve found a solid understanding of data manipulation has served me well in my work as an accountant. My primary tool for 4 years has been SAS. It is an awesome, easy to learn language that has allowed me to look like a hero on numerous projects. However, starting next week, I’m moving to a company that doesn’t offer SAS as an option.

I didn’t want to lose the edge that a good analytics package offers, so I started looking around for an open source alternative. That’s when I found “R”.

Like SAS, R is a statistics package. It’s designed for statisticians to perform t-tests, calculate standard deviations, and all the other black-box stuff that gives auditors the warm-fuzzies. However, like SAS, it also has strong data manipulation abilities. There is even a package called sqldf, that allows users to write SQL statements in the R command line.

Not surprisingly, because the name “R” is so generic, it is nearly impossible to find good tutorials online through tradition searching. So, I decided it was time to buy a book. Fortunately, Robert Muenchen from University of Tenn, has written a book for people exactly like me: R for SAS and SPSS Users (ISBN-10: 0387094172).

Unlike a lot of programming books, Muenchen’s is packed with examples and real world applications. He walks readers through a particular scenario, then shows how to accomplish with SAS, SPSS and R. It’s like a Rosetta Stone for users needing to move from one language to the other. Overall a great help and I’d highly recommend it.

April Fools Is Better With Visual Basic

Happy April Fools-Eve, my fellow practical joke connoisseurs. As someone who enjoys a good laugh at another’s expense and also happens to be a bit of a geek, I thought would post the VBA code for two of my favorite Excel-based jokes. For those who aren’t familiar with Visual Basic for Applications (“VBA”), it’s a programming language that Microsoft created which allows users to customize the functionality of MS Office. For example, I have two macros setup which create hotkeys for Paste Values and Paste Formulas, since I use those on a daily basis. VBA is really useful when used correctly.

Well, like any powerful tool, VBA can used for good and evil. That’s one of the reasons Microsoft has warning messages and sirens every time a workbook is opened that contains code. On April 1 every year, I find VBA good for some laughs. Here’s two of my favorites. (Full disclosure: The first one I created myself, but be careful as it can cause work to be lost if not used appropriately. The second I took from a VBA forum and modified to suit my needs.)

Automatically Close Excel When A Workbook Is Opened
This one is simple, but I don’t tend to use it unless I plan to tell someone how to fix. Essentially, when a file is opened, the code below will automatically close all instances of Excel and tell the user “Excel is currently experiencing technical difficulties.” Fair warning, you’ll want to do this when you’re sure all instances of Excel are already closed on their machine. If they have anything open, they could lose work and then nobody is laughing.

Excel has a standard folder that, when the program is launched, every workbook in that folder is opened and all VBA code is auto-executed. Here’s the path:

C:\Documents and Settings\[User name]\Application Data\Microsoft\Excel\XLSTART

If you create a new workbook in Excel 2007, go to Developer > Visual Basic, and copy the code below in the “ThisWorkbook” section. Save the file as .xls (2003) and then move it to the path above. Once you are ready for the madness to end, or your coworker is on the verge throwing their computer out the window, just remove your file from their XLSTART folder.

Private Sub Workbook_Open()
MsgBox "Excel is currently experiencing technical difficulties."
Application.DisplayAlerts = False
Application.Quit
End Sub

Excel Is Very Polite Today
I really like this one. It is harmless, and instantly recognizable as a joke. Only works at the workbook level, so you’ll have to hand pick the file you place it in.

Essentially, every time a user changes a cell in the target range (A1:AZ10000), a message box appears saying “Thank you. Cell XYZ has been changed.” It’s awesome. Just follow the same process as above to copy/paste the code into the VBA window of the desired sheet in your workbook.

Private Sub Worksheet_Change(ByVal Target As Range)
Dim KeyCells As Range

' The variable KeyCells contains the cells that will
' cause an alert when they are changed.
Set KeyCells = Range("A1:AZ10000")

If Not Application.Intersect(KeyCells, Range(Target.Address)) _
Is Nothing Then

' Display a message when one of the designated cells has been
' changed.
' Place your code here.
MsgBox "Thank you. Cell " & Target.Address & " has been changed."

End If
End Sub

Hope everyone has a safe and happy April 1. April Fools is always better when you’re a geek.

Mapping an MS Access Database to SAS Library

This post is for people, like myself, who assume things are harder than they usually are. Worked on an analysis yesterday where I needed to map an Access database to a SAS library so I could use the tables in some joins. My first thought was ODBC. I proceeded to setup the connection, only to find out my company didn’t give everyone that ability. Bummer. 15 minutes wasted.

Searched around and couldn’t find any other ideas. Then I thought, I’m going to try to just reference the database directly in the LIBNAME statement and see what happens. It worked. So, this post is to save people who, like myself, usually try to hard way first, only to find their way back to simplicity.

Example:
LIBNAME TEST "C:\test\database.accdb";

JoA: Dashboard Your Scorecard

The Journal of Accountancy had a good article last month on building Excel-based dashboards. It got the creative juices flowing for me, and I am thinking about integrating some dashboards into my daily work.

Modern BI tools like SAS and Business Objects can be used to automate feeds to predesigned Excel dashboards which can then be accessed by anyone over a shared network drive. The hardest (and most important) step is determining the appropriate metrics.

Dashboard reports created in Microsoft Excel are powerful, flexible and easy to design. In much the same way that an automobile dashboard graphically displays numerous measures of performance from the gas level to oil pressure, a computer dashboard presents critical data in a variety of visual formats. From this organized visual display, optimal business decisions can be made quickly and efficiently.

Excel: VLOOKUP vs. MATCH / INDEX. No Contest.

Last week, I posted a tweet declaring that VLOOKUP (the popular joining function in Excel) was for newbs and that cool kids used MATCH and INDEX. I got some interesting responses, so I thought I would take a moment to defend my position and maybe gain some converts.

VLOOKUP in Action
Essentially, both sets of functions are simple ways to join two tables that share a common field. Here’s an example of VLOOKUP in action.

As you can see, we are trying to join the customer name to a table that contains individual orders using the customer number. The function basically says, “Hey Excel, go to this array. Find the first row that has my value in the first column of that array. Return the value from “x” columns over in that same row.” Pretty simple.

The Problem
What if the key column you are using is in the middle of the array and you need to pull a value to its left? Well…you’ve got a problem. VLOOKUP will only index from left to right. If you could simply enter negative numbers in the column index value there would be no issue…but you can’t.

The Solution: MATCH and INDEX
MATCH and INDEX are two separate functions in Excel, but used in concert they can be very powerful. Here’s how it works.

INDEX requires two inputs: array and row number. (It can also accept a column number, but that’s a post for another day.) In this example, the array represents the values you would like to populate in your new table. The MATCH function is inserted into the row number section of the INDEX function to supply the appropriate row.

As you can see in the example, it doesn’t matter if the array is left or right of the needed data, since all INDEX needs is a row number. This is a powerful advantage that, in my estimation, makes it the superior choice.

GigaOM: Get Business Intelligence Ready for the Real-Time Web

GigaOM had a good article this morning called Get Business Intelligence Ready for the Real-Time Web. Here’s what they say BI 2.0 should look like. Not sure the tools exist yet to accomplish, but I can see things moving in this direction. Well designed dashboards do some of this already.

The future of BI 2.0 will not rely on server power, but on the human power of the entire organization to capture everything, convert social media into tangible business objects, and automate real-time business processes. BI 2.0 needs to help enterprises do three things:

1. Monitor everything. We need to give workers the ability to better monitor a million things at once: news, competitors, visual brand identities, consumer feedback, e-reputation, etc. (10 competitors x 10 major news searches x 5 social media sites = 500 sites each worker needs to check manually everyday just to stay industry aware!)

2. Self-organize. Flickr and YouTube’s content was considered too big to organize, until tagging came along. Similarly, we need to help enterprises self-organize the billions of clicks, searches, Tweets and reports they produce every year within their company by harnessing the tagging talents of experts within their organization. Private tag clouds, anyone?

3. Trigger processes. As we uncover trending topics within the organization, we need to enable it to automatically trigger actual business processes in real-time, like purchase orders or ad budgets.

xkcd: SQL Injection Comic

I love xkcd. This is old, but I’m slow on the uptake. I also love that a comic spawned a website on how to prevent SQL injection attacks.

Exploits of a Mom

SAS EG: Filtering a dataset via prompt manager variable

Nobody who reads this blog is likely a SAS programmer. However, on the off chance this would be useful to someone who finds it via Google, I wanted to post.

Currently, I am working on a project in SAS Enterprise Guide where I needed to allow the user to provide variable value(s) via prompt manager and then filter the resulting dataset based on the user’s input. After rubbing a few brain cells together, created a macro that does the trick. It works by embedding a loop inside of a WHERE statement in SQL. If the user selects one value the loop ends and returns that value into the WHERE statement. However, if multiple values are selected, the loop iterates until all values have been returned.

There’s probably a better way, but at least this works. Here’s a quick outline of the variables.

PRODUCTCATEGORY: This is the name of the prompt variable I setup. Replace it with your variable of choice.
IN: Input dataset
OUT: Output dataset
FIELD: Variable in the dataset on which the filtering is performed.


%MACRO PROCESS(IN,OUT,FIELD);
PROC SQL; CREATE TABLE &OUT. AS SELECT * FROM &IN.
WHERE &FIELD. IN (
%IF &PRODUCTCATEGORY_COUNT = 1 %THEN
"&PRODUCTCATEGORY";
%ELSE %DO I=1 %TO &PRODUCTCATEGORY_COUNT;
"&&PRODUCTCATEGORY&I"
%END;
);
QUIT;
%MEND;

Crypto and Backdoors

Steve Gibson and Leo Laporte did a great podcast this week regarding the Obama administration’s proposed crypto-backdoor legislation. This is same issue Schneier discussed in his post from last week. Here are my takeaways.

  • The technology for strong encryption already exists. It’s math. Bad guys will still use it. Backdoor mandates will only hurt law abiding citizens.
  • If this passes as requested by law enforcement, secure peer-to-peer communication would become illegal and the systems that support it subject to redesign. This includes corporate VPNs, Skype, and a host of other communication services.
  • Backdoors would only serve to open security holes that hackers can step through to cause problems for honest people.
  • Wired and the EFF also did pieces on the story that are worth reading. I can’t see this making it far in the legislature, but stranger things have happened.

    Update 10/6/10:
    Reread the NY Times article that started this whole conversation. The final FBI quote is so misguided that I had to comment.

    “No one should be promising their customers that they will thumb their nose at a U.S. court order,” Ms. Caproni said. “They can promise strong encryption. They just need to figure out how they can provide us plain text.”

    The whole point of encryption is that no one but the two authenticated parties can decrypt the data. If companies can decrypt their client’s data then by definition the product is not providing strong encryption. I understand the FBI’s dilemma, but they really need to rethink their position.

    CPAs should watch this issue unfold as it has the potential to dramatically change the IT landscape.