The first step towards this end was to convert the stable connectors into packages composed of synclets being managed by the common code.  This has been completed (we’re still working on cleaning up some of the patterns to get them right, but we have a start checked in that we’re playing with) and they are now ready for people to start poking at.  The current plan is to eventually migrate all of the connectors (and all the data your connectors have been collecting!) to synclet powered versions.  For the time being, existing connectors won’t be affected by any of the work being done around synclets.

The missing pieces that will tie this work together and make it easy for developers to implement their own synclets mostly surround authentication and creating a UI for managing installed synclets.  We will be implementing something like everyauth to manage authentication for synclets at some point in the near future.  This will enable us to simplify both the UI, and the implementation, for authentication and allow us to provide authentication keys to any number of synclets that would like to pull data from a provider.

What does this mean to developers hacking on connectors today?

Not a whole lot just yet.  We want to spend some more time ensuring that we have this pattern right, and connectors will continue to exist exactly as they are today during that period.  If you’re feeling super adventurous, feel free to poke through some of the code in lsyncmanager, and look at the new synclets that are now provided with the connector code for Facebook, Twitter, Github, Foursquare, and Google Contacts.  Once the authentication and UI pieces have been baked into the project, we’ll write a more detailed post describing exactly how you would use those pieces to build synclets.

Synclets

These are a break out from the connector work to try and figure out better patterns around collecting data from outside sources. If you are a connector author then you definitely want to check out the code and give the wiki page a read.

Search

We’ve been exploring different patterns of using search within the Locker, which Eric wrote about, and we’ve finally got a new bit working. The CLucene implementation has come alive! It’s still basic, but it builds a single query point for everything in your Locker. You can try it out with the Locker Search app, but it still needs some UI love.

Photo Viewer

Tom just landed (literally as I’m writing this!) an experimental photo viewer that looks fantastic. So go get your photo connectors setup and quickly browse them with this great viewer.

Discovery 1:  Lucene is a pretty amazing information retrieval library

Lucene is an open-source Apache project that handles information storage and retrieval, specific for the uses of searching and information retrieval.  It is actually just a library, however, meaning we can’t easily slide it into the existing Locker codebase.

A little detail about Lucene, and why it’s so powerful.

• Lucene implementations have scaled up to the terabyte range of storage.  This fares really well as each user’s Locker may grow quite large over time, and we don’t want to outgrow an architecture if possible.
• Lucene has clearly-defined concepts of text processing, both when indexing–placing information into the system, and when querying–pulling information out of the system.
• Lucene also has some subprojects that may turn out to be very helpful for Locker users. One in particular is Tika, which can extract the textual contents from binary files such as PDFs and Word documents.  This is super helpful if, for instance, you wanted to not only search across your entire Gmail account, but also wanted to search within any attachments others have sent to you.
• The Lucene library has been battle-hardened over several years of active development and lots of implementations in production.  Some very important thoughts have emerged around scaling Lucene, based on actual users pushing it to its limits.  This gives us a lot of useful information, and shows us what not to do when adding Lucene to Lockers.

In short, we were pretty sold on Lucene as the search platform of choice.  However, we weren’t ready quite yet.

Discovery 2:  Lucene has lots of implementations, and each of these has tradeoffs

If only we could download and install a simple library and have full-search Lockers that easily!  No, unfortunately, there are several implementations of Lucene, and we want to choose the best based on the unique requirements of Lockers.

First, we tried elasticsearch, which is a very robust implementation of Lucene, along with capabilities to allow it to scale to MASSIVE sizes.  It also has a very robust method for indexing and querying the information store from pratically any client.  We prototyped this one first, knowing that it was large, and probably overkill, but we could begin to get search results in a day or two, because of its maturity.

Sure enough, a couple days later, we were searching across the Locker, as you can see here:

However, this required every Locker user to run a full instance of elasticsearch alongside their Lockers, which is a heavy requirement. However, we loved how easy it was to get going with useful searching, so we committed that code as the Locker app called “Search”, and forged ahead.

Discovery 3:  Lucene isn’t only a Java library

How to get Lucene embedded within the Locker?  That was our next task.  We wanted something lightweight, fast, and embedded so we didn’t have to add yet another external requirement.

That’s when we learned that a group had ported the Lucene library to C++, and the project’s called CLucene. This was fantastic news for us, as we figured we could just wrap clucene into a native Node.js module, and it’ll be extremely fast and small, but still retain the power of Lucene.

Discovery 4:  CLucene looks great, but it’s based on a much older Lucene version

Daaaaamn.  We were so close, and it turns out that CLucene is missing some of the features we really wanted to use from current Lucene implementations, specifically geopoint support and updating existing documents.  Now we needed to look closely at CLucene to see how much we need to implement.

But first, like most of our projects at Singly, we applied the mantra “Make it work.  Make it right.  Make it fast.”  So before getting it as full-featured as we desired, we first just wanted to make it work inside the Locker natively.  This turned out to be a good learning experience for us, as we now know how to write native Node.js modules, and that opens up the massive amounts of C and C++ libraries out there to us.

Enough technobabble, let’s see what this sucka can do!

Okay!  Here are some queries you can run on your Locker:

• “eric”  - Find everything that has the term “eric” in it, case-insenitive
• “eric~” – Do a fuzzy search for “eric”, meaning be a bit forgiving about misspellings.  This will return back data that contains terms like “erick” and “erik”, as well as “eric”.
• “+lockerproject singly” – Do a search for data that definitely has the term “lockerproject” in it, but may or may not have the term “singly” in it.
• “+lockerproject singly^5″ – Do the same search as above, but for those records that do have the term “singly” in it, boost those scores much higher, so they show up closer to the top of the search results.

What’s next?

Well, this is where we need to add functionality to CLucene that it did not already have.  Specifically, we need to be able to add all the different fields that are internal to Locker records.  For instance, for a record representing one of your contacts, you should be able to search for first name separately from last name.  Right now our Node.js CLucene module does not support that.

We also want to add back in the ability to update existing records, so your data is always searchable and retrievable, regardless of how many times you resync your connectors.  Also, being able to search distances from a geo location such as a Foursquare checkin is very high on our list as well.

We’re working hard to bring useful features to Lockers across the web, and search is going to be a core part of that.  We are always happy to hear your thoughts or answer any questions you may have.  We are regularly available in the Freenode IRC room #lockerproject, as well as via e-mail and github.  We welcome your feedback!

Dashboard

We’re continuing to work towards the best dashboard interface possible and have made some more tweaks to the layout. We’re working on the overall first run experience and expect that code to start landing soon.

See the full gallery on Posterous

Photos

The photos collection has started to have some real meat thrown on its bones. The Twitpic and Facebook connectors are feeding it all your lovely photos that you can currently view in the Hello Photos app. It’s a bit raw still, but it’s working and plenty more exciting visualizations coming in the pipe.

OSCON

Jer, Eric, Matt and myself (temas) attended OSCON and it was a fantastic and productive trip. I’ll let Eric’s writeup give you the details.

Lockerballs

Part of our OSCON prep was to create a single precompiled tarball that everyone can use to quickly get started with the Locker Project. They have been dubbed Lockerballs and can be found on the website. A giant shoutout to Forrest for the work he put into getting them done and maintained. The Lockerball is a clone of the git repository so once you are ready to start hacking you can git pull and join the fun.

The plan is to migrate to full virtual machine images in the near future.  We’re working on cutting out as many roadblacks as possible so more people can be testing and working with us.

Synclets

On the forefront of new code we have Synclets. These are a break off from core connector code to simplify and contain the actual syncing logic. These are a great move to a clean and ordered system that is headed up by Chris and Jer. If you’re writing connector code check them out and get us feedback to help their early development.

The Concept of Xen micro instances for running Lockers:

We spoke with Jeremy Fitzhardinge from XenSource/Citrix, regarding running Lockers in micro instances of Xen.  Through a lot of discussion, it sounds like it’s at least feasible in theory that very small Xen instances could indeed be used.  We also discussed the possibility of running Xen inside of Xen (to run Xen instances on a platform such as AWS), but it appears that this doesn’t peform well, understandably.

Plug Computers:

I attended a session that talked about the current state of small “plug” computers–defined as those types of computers that are small enough to simply plug into a wall.  These usually range from very small thumb-drive sizes up to a large wall wart or slightly larger.  The DreamPlug was one being demoed, and could be very interesting as a platform to use as a Locker appliance.  The good news is that besides the various networking capabilities of the DreamPlug, it also has an eSATA port on it, meaning it’s simple to plug in a large, high-performance hard drive for lots of storage.  Storage tends to be one of the weakest aspects of plug computers, and being able to add storage as necessary makes this very interesting.  Its price, when released, is said to be around $99.00 USD.

Here are the DreamPlug specs:

• CPU – Marvell Kirkwood 88F6281 @ 1.2GHz speed
• Linux 2.6.3x Kernel
• 512MB 16bit DDR2-800 MHz
• 2MB SPI NOR Flash for uboot
• 2 GB on board micro-SD for kernel and root file system
• 2 x Gigabit Ethernet 10/100/1000 Mbps
• 2 x USB 2.0 ports (Host)
• 1 x eSATA 2.0 port -3Gbps SATAII
• 1 x SD socket for user expansion/application
• WiFi 802.11 b/g
• Bluetooth 2.1 + EDR
• Audio Interfaces
• 5V3A DC power supply

Another plug computer that came across the radar is the Raspberry Pi.  This is a tiny computer, yet is ARM-based and can run Linux.  It’s does not have a lot of storage support other than SD cards, but could be extremely flexible in being able to run one or more connectors and stream data back to a central Locker instance.  I can see this being used with the Arduino stack to get cheap, realtime sensor data from the environment and into Lockers.  Its price is $25.  (Cheaper than the Arduino Uno dev board!)

Here are the Raspberry Pi specs:

• 700MHz ARM11
• 128MB of SDRAM
• OpenGL ES 2.0
• 1080p30 H.264 high-profile decode
• Composite and HDMI video output
• USB 2.0
• SD/MMC/SDIO memory card slot
• General-purpose I/O
• Open software (Ubuntu, Iceweasel, KOffice, Python)

Handling Locker Storage Safely and Quickly:

How to safely scale and store Locker data has been an ongoing discussion within the Locker Project.  Several options are on the table, and a lot depends on the requirements at hand.  For instance, for Singly-hosted Lockers, we will need something that we can host on insecure platform-as-a-service providers such as Amazon AWS.  For this, something like Tahoe LAFS looks like a great contender.

Other requirements that span both Singly-hosted Lockers and self-hosted Lockers are the capabilities of authorization to access, proof of change of data, and versioning of previously-changed data.

So it was interesting to have met up with Brad Fitzpatrick from the Memcached/Danga/LiveJournal world, to chat with us about his new project Camlistore.  He gave us a very quick rundown on various features.  One of note is the key signature of any changes made to the filestore, which allows easy confirmation of who changed what and when.  As it was described to me, it sounds like this same functionality also allows only approved users to view sets and/or subsets of the data.  Lastly, the versioning support that it has could prove to help us with our versioning issues as well.

One use case I found quite compelling was Brad discussing the possibility for a Camlistore (and by extension, a Locker) user to provide a subset of data that other people can write to.  For instance, you could provide a set of photos that you took while attending a party, provide write access to those particular photos to a group of friends who were also at that party, and those other users could add data–such as comments or tags–to your dataset, all using the Camlistore functionality.  This peer-enrichment idea is super exciting to me.

Locker-wide Search:

I met up with Tyler Gillies, who wrote the node-lucene module for Node.js.  This module wraps up the CLucene library and exposes it to a Node instance.  CLucene, for those who aren’t familiar with it, is a C++ port of the Java Lucene library–itself a very mature and powerful information retrieval library.  Internally, Singly has a Locker-wide search application already running, but it’s a prototype and requires the Elasticsearch Lucene implementation to run.  Elasticsearch is amazingly powerful, but it is also very resource-intensive.  We need something smaller, leaner, and faster, and CLucene fits the bill well.

The last night we were there, Singly put in a Locker BIrds of a Feather session where a bunch of us hacked on Locker things together.  I was able to work with Tyler to get up to speed on the state of node-lucene, and to begin contributing back to it with some more features we’ll need for full Locker search.  I’m super excited to get advanced search capabilities in the Locker soon, such as the following:

- Find all of my contacts from the contacts collection that have the e-mail address containing the term “singly.com”

- Find all of my geographic data from any connector or collection for the user “Eric Jennings”

- Find that link I visited recently that talked about the Google V8 garbage collection method, can’t remember if I read it on my phone or on one of my machines

If we do search right, any of thes types of search queries will be available to any application, collection, or connector.

That’s it for the OSCON trip.  Several of us were able to catch up with people we hadn’t seen in years, or have never met other than in IRC.  For those we met, it was great meeting you!  And for those who weren’t able to go, we hope to meet up soon!

Over at Singly, we were curious what types of data people most cared about, and why. As an experiment, we surveyed 179 people, at a cost of twenty cents per person. Filling out the survey took on average 2 minutes and 13 seconds, meaning the effective hourly rate was $5.414. I would have expected more results if I offered more money for the survey, but this was just a small experiment.

The question was “What is your most important personal data?” users had to pick either:

• Contacts
• Messages
• Events
• Check-ins
• Links
• Photos
• Music
• Movies
• Browser History

Not surprisingly, Photos won at a whopping 67 votes (37%). Here are the full results:

What was interesting is that people ranked Browser History, Links, Events, and Check-ins so low. Do people not care about where they went? Is this data considered stale to most people, and therefore irrelevant? I personally believe I can create a lot of value from Browser History and Check-ins. For example, what websites are my friends going to that I’m not? Also, what places should I be going that I’m not? These are just a couple of ideas.

This survey validates what I believe we already knew, that Photos, Contacts, and Messages were important collections to create. More importantly though, the second question “Why is that data the most important to you” explains *why* people feel are so passionate: (word cloud from all answers)

Photos

“Photos are most important to me because they preserve mine and my family’s history. They are a way to show younger generations deceased family members they may have never gotten an opportunity to meet. The record both happy and sad memories, which are a part of every family’s existence.”

“Many of my photographs span back from childhood, thus they’re irreplaceable. Every other option on the list is unimportant, as they’re replaceable and don’t hold sentimental value.”

Contacts

“My contacts are the most important data because without then i can’t reach my family and my friends. I have all my costumers contacts and if i loose those contacts i can’t work. I make regular backups of my phone and email contacts.”

“Friends and family are important to me.”

Messages

“These days we all do most of our work through electronic communication. So most of my transaction details, confirmations, decisions taken, and documents, come and go through email. So messages are most important.”

“It’s important to me because I am a human being, and like every human being before me and those to come after me, I have secrets. These can be in the form of affairs, a half life (think LGBT) and other more sinister varieties. Secrets that no one but the people I choose, should know about, this is why this personal data is the most important to me. Data which if exposed, parts of my well built artificial life to satisfy a conservative society and family would come crashing down around me. I am presuming you also mean emails when you say messages. This is why this data is the most important for me. I hope I have been able to help you, thanks.”

If you would like to read on, the raw data is available at: 
https://spreadsheets.google.com/a/singly.com/spreadsheet/ccc?key=0Ak8IPGG6Z4dOdC1ROHQwRkhUMEtPQTNkNXFoMnQzcEE&hl=en_US#gid=0