When you're serving static files...

Never link http://yoursite.com/foo/bar/x to:

A) plain http://static.yoursite.com/foo_bar_x.ext

B) Never use plain hashes of the ID. e.g. .../hash_of_id_here.ext

C) Borderline, but even static salts are bad: .../hash_of_id_plus_'foobar'_here.ext

Basic is, use a variable salt for every item. It can ever be public info, like the title, creation time, or anything that varies sufficiently per-item. That's the beauty of salts.

Or, alternatively, as better people than I have mentioned, use bcrypt with a set difficulty value, and all your rainbow table type fears go away.

Emacs...

Biggest reason to switch to emacs I've found so far is that the commands work by default in bash, too.

Code on the Road: Navigating The Minefield that is Visual Source Safe

Code on the Road: Navigating The Minefield that is Visual Source Safe

Such a no-brainer. Why do you even try VSS to begin with? Get the Subversion plugin, grab SVN, and 99% of these things go away.

If that's not enough, git, darcs, bzr, and mercurial provide reasonable distributed solutions. Miles ahead of VSS (Actually, three generations, but who's counting.)

Seems to me, if you're really running an 'ex-pat' office, you'd want distributed. That way you can all trade changes around without having to fight conflicts. Best part is, you can stage to production without branch/tag.

Basically, there's no excuse these days for using an old VCS. Or, dare I say, a Microsoft VCS.

On convergent evolution, and the invocation of holy wars.

Python and Ruby are the same thing.

There, with that out of the way, let me explain myself.

I was looking at a post about scene carving image retargeting which thoughtfully includes a link to his python implementation of scene carving all nicely packaged into git.

So I grabbed it, used emacs to import it into a new file, and started converting it to Ruby, since that's the language which is hitting my happy button right now (Haskell is for when I want to hurt myself). (NB said conversion I plan to post here, or at least, link to SVN for it.)

Take a look at an excerpt:

class CostMatrix(ndarray):
        def calculate(self, energy_map):
                if not energy_map.shape == self.shape:
                        raise Exception, "Wrong shape"
                (h, w) = self.shape
                self[0] = energy_map[0].copy()
                self[0] = self[0]
                for y in range(1, h):
                        for x in range(0, w):
                                bestcost = inf
                                bestx = x
                                for dx in range(x - 1, x + 2):
                                        if dx >= 0 and dx < w:
                                                if self[y - 1, dx] < bestcost:
                                                        bestcost = self[y - 1, dx]
                                                        bestx = dx
                                self[y, x] = self[y - 1, bestx] + energy_map[y, x]
                self._calculated = True

        def _get_max_index(self, row, startcol = 0):
                maxx = startcol
                maxval = self[row, maxx]
                
                for x in range(0, len(self[row])):
                        if self[row, x] > maxval:
                                maxx = x
                                maxval = self[row, x]

                return maxx

        def find_shortest_path(self):
                (h, w) = self.shape
                
                x = self._get_max_index(-1)
                path = [x]
                for y in range(h - 2, -1, -1):
                        bestcost = inf
                        for dx in range(x - 1, x + 2):
                                if dx >= 0 and dx < w:
                                        if self[y, dx] < bestcost:
                                                bestcost = self[y, dx]
                                                x = dx
                        path.append(x)

                path.reverse()
                return path


        def get_image(self):
                scaling = 0.03
                (h, w) = self.shape
                im = Image.new("L", (w, h))
                im.putdata(self.flatten() * scaling)
                return im

Now, if you're a python person, that should be fine. But what if you're a ruby person? That looks like ruby, where someone added in a lot of colons, and didn't remember their end tags. Oh, and someone's using paretheses oddly.

So, having converted those things, I'm confronted with a syntactically valid chunk of ruby code. It no longer throws parse errors.

This is pretty mindblowing, to me. Maybe it's something that's long since been obvious to the old hands...

Anyway, this makes me wonder why Python and Ruby aren't implemented on the same core compiler/interpreter. I know Microsoft is doing something akin to this with their Dynamic Language Runtime, but why aren't the Ruby people stealing like mad from the Python people, and vice versa?

That said, this, to me, is only the midway step between Python and Haskell. Ruby will probably take a week to do anything fun with it, so, much as I might like to throw up a free image resizing service, I'm thinking I'd rather do it in HAppS, where at least it will be fast.

Any thoughts? Am I an idiot for not seeing this already?

PS there's also another implementation of scene carving based resizing that I've been looking at.

Lunar Eclipse

Taken last night. If I had a better camera, I'd have taken Mars during the eclipse too.

Eliciting rational thought on the internet

Ok, this is very much a work in progress at the moment, I'd appreciate feedback a lot.

I'd like to take some time today to write about the way I interact on the internet. I'm doing this partly as a way of clarifying to myself the goals that I'm pursuing. Another part is, when people accuse me of being deliberately obtuse, argumentative, or 'debating unfairly', I can point them here, as not a means of excusing myself, but rather a means of explaining the reasons. Finally, I think the world would be a better place if people examined the way they interact with others more often, so I'd like to be less of a hypocrite by spending some time in the pursuit of that goal.

Firstly, let me lay out my goals:

Learning.

This is always high on the list. Everyone teaches you things.

Teaching others.

Again, I'm not sure this is as noble as all that, given that most people don't like to be taught a lesson even if it's good for them.

Developing communication skills.

This is something that I, as with, I think, most technical people, have problems with.

Getting others to think before they comment.

I'd like to see more people think to themselves 'No, that's silly, I shouldn't say that' or 'Hmm, is that really true?' before they spout off about the topic du jour.

So in pursuit of these, I do a few 'not so nice' things, though I believe, as is the call of petty tyrants throughout the ages, 'the ends justify the means'.

Feigned ignorance.

This is probably my favorite thing to do. Most people, in the course of explaining what 'everybody knows' will end up realizing that they don't have a firm grasp on the topic at hand. I end up doing more research myself this way...

Questioning of basis.

I've been accused of being a dirty debater for asking people a pretty simple question: "Why do you think that? Do you have any evidence or proof?" People come up with the most marvelous sources when you ask them this, especially when they're experts in the field.

Employing logical fallacies.

This is almost a test: Do you recognize when someone is using these? I wrote about one that always bothers me in Sunk Cost: Fallacy and Dilemma, but there are many more. People don't seem to realize that this is one of the key tools of a demagogue. Immunizing people against them should be one of the key tasks of education, but I'm fairly sure my school didn't even mention them once.

Playing Devil's Advocate.

This is the classic trick of the troll. Take a deliberately extreme position, and defend it well. In doing so, you're far more likely to solidify people's current beliefs than change minds. In a lively debate, often both sides come away better educated, perhaps having gained a modicum of respect for their opponents.

(This is one of the key problems with politics now: nobody can put themselves in the other person's shoes, everything is a life or death issue)

Anyway, I've got to sleep now, but there it is: When I talk to people on the internet, it's often with goals in mind.

From a friend's recommendation letter...

New post on this subject: How to write a letter of recommendation

---

I was helping a friend write a recommendation letter and came across these gems:

- "Self Goal Oriented"

- "Most plausible vehicle for my goals"

- "Practice my compassion"

Yep, lessons in what to avoid when talking to literate people.

Also, I'm still alive out here and plan to have another post up with real content soon.

Trendlines and Musings

It seems that there have been several general and successive trends online over the last few years, each driving a certain type of success story (and yes, some overlap exists):

1. Search (Google, Ask.com, Y!, tagging on Flickr, etc)
2. Share (del.icio.us, Digg, YouTube, Reddit, etc)
3. Create ... this one is still in its nascent phase, but is cropping up more and more, often in conjunction with the first two
4. ???

My lines of inquiry here are: what is the eventual trajectory for the "create" modality, what comes after it, how can these work together, and what technologies/techniques might offer the best fit in pursuing these new projects?

Create
The creation modality of the net is nothing truly new. We've long seen services that allow users to create and customize their own home pages, and blog services that allow people to post and host their own content.

This creation trend has been expanding into other areas more recently, pulling more varied media types and methods into the fray. Delicious:designer, for example, is an intriguing new tool that gives a pretty slick graphic design interface to its users. There have been spin offs of the "picture a day" internet meme that use Flash to pull images directly from connected web cams into the internet application, automating the creation of an image stream. There are similar services available for creating music mashups or playlists to share with friends (both through links and via p2p, but the discussion of technology will come later).

There have also been some pushes to create services that allow people to create dynamic content, like web games and other small apps, all through simple interfaces. I know that HAppS has some interesting potential here, as do things like Flash, Flex and possibly even Laszlo.

Looking at the progression of create-style web apps, we see a trend of offering more and more control in creating more rich content. The examples that seem to be most successful, in my experience, are those that offer the simplest interfaces, and the simplest hooks to allow people to integrate their creations into the search/share spaces.

Search/Share
Uptake of content is the name of the game. Anyone can "publish" these days. Getting content online takes about half a moment, and even less thought. Getting that content into the hands/eyes/minds of others is the trick. Getting that same content to your desired audience is the holy grail of search/share, and it's why companies have been able to pull in hundreds of millions of dollars in that space.

As more dynamic content is created, there is more noise added to the search/share channels for those media types. Let's face it - it's not all good content that's being produced. So how do end users find good content - and the converse - how do content producers get their content into the channels that will get to good audiences? Well, most of that discussion is for another time and place, but from a create-mode product view, there are several emerging strategies that seem to be doing well:

• the aggregate and vote system (ie, Digg and Reddit) allows a wide pool of users to filter material, giving exposure to a lot of content, but pushing a certain amount to the top
  
• the "email to share" system allows people to push interesting content directly to their friends (making this *easy* and non-intrusive is key, though)
• social network messaging systems allow people to post to a focused group of consumers, who can then propagate the message further through their networks if they find the content appealing

There are more methods, to be certain, and it will be interesting to watch them as they continue to evolve.

Technologies/Techniques
Speed. Simplicity. Engaging.

These concepts have really been at the forefront of the current trend cycle. Agile development and rapid application development help get things from concept to reality in short, repeatable cycles, allowing things to grow organically and respond to feedback quickly. Simple focus - doing one thing really well - has made the use cases for products really clear. This lets their utility shine through, promotes ease of use, and keeps things focused on what works and what's needed. Finally, keeping people engaged has been critical. This is no longer a broadcast world. Commentary, dialogue, edits, versions, iterations - these all form the critical feedback that keeps the system evolving.

The tool sets and technologies that really respond to these things have definitely seen a lot of attention in recent years. Agile has taken off like a rocket. Rails has been going gangbusters, maturing by leaps and bounds. Languages with high ability for concurrent execution have been getting more attention. Tool sets that allow for mixed online/offline utility have been popping up like mushrooms.

... to be concluded in next post.

Travelling Salesman Problem: Introduction (in Haskell)

Here's the original link.

The task here is a setup for some stochastic methods of solving the Travelling Salesman Problem. I'm doing it in Haskell since my skills here need work.

basic.lhs:

Set up a coordinate string. These are the cities in our routes.

> coordinates :: [(Float,Float)]
> coordinates = [(0,1),(1,2),(2,3),(2,4),(0,6)]

This is a simple function that takes two 'cities' and returns the floating-point distance between them using the Pythagorean Theorem

> distance :: (Float, Float) -> (Float, Float) -> Float
> distance (x1,y1) (x2,y2) = sqrt ((x1-x2)^2 + (y1-y2)^2)

This allows us to determine the actual length of a list of cities ('tour'). Note that it's not circular, unlike in the linked article.

The 'let' statement looks up the city in question from the list above and inserts it into an equivalent list.

zipWith is like map, except it takes a two-argument function, and two lists of arguments which are 'zipped' together into a result list.

> tourLength :: [(Float, Float)] -> [Int] -> Float
> tourLength pairs tour = let coords = map (pairs !!) tour 
>                         in sum $ zipWith distance coords (tail coords)

A wrapper, if you'd like to have all tours be circular.

> tourLength' :: [(Float, Float)] -> [Int] -> Float
> tourLength' pairs tour = tourLength pairs (tour ++ (head tour))

The main testing function, which prints the specified tour's length

> main :: IO ()
> main = print $ tourLength' coordinates [2,1,0,3,4]

One thing you'll notice here, is that I'm eliding the matrix calculation in the original source. One of the nice things about Haskell is that a later date, should I choose to do some memoization, there's very little work to be done. Only one of these functions would need to change.

Another nice thing is that if a given pair of cities is not ever used, Haskell (due to it's laziness) won't ever bother to actually calculate the value, even if I do go back and add memoization.

There's more to this, especially the shuffling functions (I'll get to introduce Monads!). That'll have to wait for another day soon.

mimes.

Mime shouldn't be a capital crime. The first time.

Business metrics

A hypothesis for you: it seems that the more time I have to spend killing time via surfing the 'net at work, the shorter the time I can expect to remain employed there.

Maybe someone should build a small aggregate logging system to determine company health this way...

Alt Framework Hackfest

We here at IT appreciate the diversity of web frameworks and methodologies that have arisen or gained popularity over the last few years, and we've had our filthy paws over and into a few of them. Some we like more than others, and some of us are more vocal in our advocacy than others.

What I'd like to propose to my co-conspirators here is that we have a little hackfest with some of our favorite web frameworks over the next month or two, and see what develops. I'd propose some similar goal to be developed in each framework, but that might be too constricting. I'll let you guys weigh in before we kick it off, just to get consensus, but I'm guessing spinning out a blog in each shouldn't be too hard, and it would allow for embellishment as the frameworks and time allow. (note, I know we all have crazy schedules right now, which is why I think a month or two sounds good... correct me if I'm wrong)

Frameworks I'd like to see included:

• merb (with maybe some Rails for support... if required)
• HAppS
• seaside
• kepler, anyone?
• erlang
  

I think it would be great to show some of these frameworks side by side, performing similar tasks. Sure, the validity of the comparison would be limited by the scope of the project, but I think it would be nice to see functional apps from each, if we're going to keep hearing the same evangelism about Framework X or System Y every damn time I log on.

Cheers.

Google Gears

Google is at it again, this time maintaining its position at or ahead of the curve with Google Gears.

What is it? It's the latest in a great trend toward allowing for creating hybrid web apps that can run on- or off-line. Google Gears is by no means first to the party, however. Joyent was out the gates early with Slingshot, which we had the pleasure of getting a look at up at RailsConf. Adobe is also trying to get in the game (from an admittedly different angle) with its work on Apollo. Add in to this mix the browser-as-dev-platform path that Firefox 3 might be taking, and we can see that the world is moving very rapidly toward a synchronizable, online/offline world.

So why do we care about Google Gears? I'm sorry, that was a stupid question. We care because it's Google, and they have the intellectual resources to really solidly address this path, coupled with their penetration into just about every technological market. Also, they're following the grand tradition of their code projects, and opening a lot of code base for use by developers. The APIs are documented and available, and they're encouraging devs to play with the beta.

Gears provides the following basics, as quoted from their site:

	LocalServer Cache and serve application resources (HTML, JavaScript, images, etc.) locally
	Database Store data locally in a fully-searchable relational database
	WorkerPool Make your web applications more responsive by performing resource-intensive operations asynchronously

This should be enough for any web app developer to get off the ground and running, building relatively responsive RIAs... on the desktop.

Ok, enough drooling over docs. I'm going to go dig into it.

I'll probably post back later with impressions, but feel free to throw your comment in, too. I'd love to see how this is responding with different frameworks and on different platforms.

Sunk Cost - Fallacy and Dilemma

First of all, I think it's pretty funny that the first substantial post here is me, talking about project management stuff, but it's true.

On to the good stuff:

---

Sunk Costs
The definition of a sunk cost is simple, but with some subtle points.
Sunk costs are the unrecoverable past costs of an endeavor.

Examples:
Buy a sandwich and eat it. Sunk cost: sandwich price + time spent.

Buy a book, read it. Sunk cost: Difference between the retail price and the resale price, plus the time spent.

Now that that is out of the way, we can talk about why they matter.

---

Sunk Cost Fallacy
This is the formalization of the old adage about not 'throwing good money after bad'.

Basically, people have an irrational tendency to count sunk costs when they make a decision. This is pretty obvious to see in the tendency to never quit on a failing project, even if it's the overwhelmingly right thing to do.

As wikipedia says, you don't see many half finished bridges.

---

Sunk Cost Dilemma
Here's the really interesting part: due to sunk costs, always doing the right thing (in a naive setting) can result in failure.

Say you estimate a time and date for a project, sign the contract, and start work with the expectation of plenty of time and resources.

Midway through, your star takes sick. You have to decide whether to keep going or pay someone else to finish it. You figure there's still time, so you keep going.

Now, what's the right choice if you run into another problem? If you look at it naively again, odds are good you'll keep going in hope of that payout, even if you'll lose a little money.

The right answer to this problem is twofold: First, you assess the likelihood of the outcomes, and secondly, you take into account possible future costs of each outcome.

---

Anyway, to relate this back to the realm of technical things, this is one of the foundation principles of Agile/Lean methods: the more risk and the more competitive the market, the shorter your feedback cycles need to be.

I see this all the time, with feature creep and bloat. Once you build it, it's a sunk cost, whether or not it's at all useful to anyone.

In many cases, the rational answer to feature creep and bloat is to throw many things out or start over, but people hold on to the existing features because they think the money spent on them will be 'more wasted' if they are removed.

Steve Jobs On Web 2.0

Look at all these new companies in the Bay Area, all these supposed “tech” companies. God knows what they do, but it’s all some variation on the same theme and they all have names like Zizzl and Drizzl and Bizzl, so you can’t tell them apart. Can’t these kids think of anything original? Apparently not. My theory is it’s because they’ve never used psychedelics.

Fried Toast

Since the title of the blog is Illicit Technology, I thought I'd give my favorite example of rebel engineering, a spidercar from Burning Man.



I was actually just out of sight when this was shot, off to the right, about where the blog sidebar is.

Frist Poast

Hi all, first post on the new blog. Hopefully this will be a blog about tech and related idiocies.