Updates from quartzjer Toggle Comment Threads | Keyboard Shortcuts

  • quartzjer 7:46 pm on May 1, 2012 Permalink | Reply
    Tags: , ,   

    Next steps for the Locker Project 

    Since we started this journey, the Locker Project and Singly have progressed side-by-side, with Singly as the hosted experience that sponsors the Locker Project.

    After opening Singly’s Locker hosting to developers and getting lots of feedback on all of the possibilities that a hosted Locker could enable, the resounding theme was that developers want to use the API first and foremost.  Based on that, the Singly team has been working on an effort to bolster the API aspects of the codebase so that it can support apps at a large scale.  There’s been excellent progress in creating a cross-platform, cross-service API that provides merged, normalized and de-duplicated data on which apps can be built.

    So now there is the question of how best to support and improve the open-source Locker Project, since it encompasses more than just an API.  It’s an effort everyone in this community cares deeply about and has helped create a shared vision for, so we’re calling upon you to help decide what the most powerful direction for the Locker Project should be.

    Please help take the time to fill out this survey, and most importantly, thank you all for your ongoing support and hard work!

     

  • quartzjer 1:01 am on January 22, 2012 Permalink | Reply  

    Code overhaul of the “map” 

    This last week landed one of the larger sweeping updates to the locker core codebase, cleaning up a mountain of tech debt that had crept in over the last year in how the various services (apps, collections, and connectors) are loaded+handled and also a solid step forward towards our Architecture Initiatives.

    It’s one of the most exciting advancements yet since it’s enabling a very dynamic and rich growth of the apps and connectors available: everything is installed on demand using npm into a running locker. The core router manages the process so that there’s almost no friction for developers creating new experiences or connecting additional sources of data.  Locker owners now always have available instantly the best apps being created.

    There’s some more detail on the wiki’s Registry page for those doing or planning any development to help with local testing out, it enables a local dev locker to pick up changes live using the upsert command for faster turnaround and more flexible dev patterns.

    Also, we’ve nearly got node v0.6 support in master finally, some “me” profile support to make it easier for collections to differentiate produced vs. received data, and some music and fitness connectors showing up! 

     

    • Duane Johnson 10:10 pm on January 27, 2012 Permalink | Reply

      Way to go guys! Thanks for writing these updates as you go along. I’m itching to dive in but waiting for the barrier-to-entry threshold to be low enough that I believe it becomes an effective use of my (part) time. So these blog posts are very helpful to gauge whether or not the time is right for me. And after this post, it almost feels right :)

    • quartzjer 11:08 pm on January 27, 2012 Permalink | Reply

      Thanks! The barrier is dropping fast, we’ll def keep updating :)

    • Akshaya 1:06 pm on March 28, 2012 Permalink | Reply

      Hi, does locker code available @ GitHub supports Node v0.6 now?

      • Matt Zimmerman 3:35 pm on March 28, 2012 Permalink | Reply

        Yes it does!

        • Akshaya 3:37 pm on March 28, 2012 Permalink

          OK thanks! Then there’s big need to update the documentation coz it still says v0.4.9.

  • quartzjer 11:36 pm on January 3, 2012 Permalink | Reply  

    Architecture Initiatives 

    There is so much incredible progress lately in the codebase and a tremendous amount of fun stuff yet to do, that I thought it would be helpful to outline for everyone some overall architecture thoughts of my own that help guide my excitement and interest :)

    Data Storage

    Include tools for the codebase manage data storage internal to itself and expose utilities for the owners to control and add storage endpoints:

    • visible to owner (where is my stuff stored)
    • ability to connect storage endpoints:
      • dropbox
      • s3
      • wordpress
      • personal desktop/laptop
      • google/icloud/amazon?
    • have the locker incrementally back itself up to the connected storage
    • let apps have the ability to direct raw copies of things (like photos) to any storage endpoint

    TeleHash

    This work has been ongoing for a while and is now finally going to start surfacing some basic functionality, a new lightweight node implementation is in progress with the intention of being integrated into the locker core for uses described below.

    Identity & Sharing

    Have the locker core manage and be aware of the profile data of every connected identity/service, and then enable the signing of those identifiers so that they can be shared via TeleHash in a verifiable way.  Enable apps to request connection/sharing methods with anyone that will have a level of verification/trust based on this, with direct encrypted verified peer-to-peer sharing being possible.

    • improve the contacts collection with edit-ability features
    • build basic permissions system for sharing things in your locker with others
    • experiment with caja/capabilities-based access logic (share bits of code)
    • include safe/simple experiences explaining what is being shared, how secure, and to who
    • strong access logging of anything shared

    Routing Mesh – Core as a personal VPN

    The core ideally goes away entirely, leaving each component of the locker to attach to a shared mesh that knows how to send/receive with each other.  This enables the locker to be essentially a personal VPN with all the components that talk P2P using REST+JSON, across all of your personal devices and any hosted pieces.  This builds on TeleHash, key management, pairing, network path detection, and building out supporting software for all the desktop/mobile platforms, a lot!

    You can find (and contribute to) even more in the original raw brain dump style document on google docs. Looking forward to an incredible 2012!

     

  • quartzjer 6:11 am on December 20, 2011 Permalink | Reply
    Tags: ,   

    Cranking away! 

    We’ve not used this blog very much recently but would love to get back into some regular use of it again :)

    In catching everyone up, the codebase has been moving *very* rapidly (https://github.com/LockerProject/Locker/network) and we’ve got a singly-dev mailing list starting to get a little traffic (http://groups.google.com/group/singly-dev).

    There’s a lot of open pull requests (https://github.com/lockerproject/locker/pulls) landing this week as well, including a new search backend (don’t need clucene and cmake anymore, uses sqlite’s full-text-search which npm fully installs), soundcloud, gowalla, a generic synclet auth system, a timeline collection, and some post-to-service experimental support!

    Hope to see you all soon in #lockerproject on freenode, on the mailing list, twitter, here, or in a pull request! ;-)

     

  • quartzjer 1:50 am on June 16, 2011 Permalink | Reply  

    Locker Arch in a Nutshell 

    We’ve had some various pieces of the overall architecture documented both in the codebase on and the wiki, but I thought it would be good to refresh and highlight the overall design here and have a new living master copy in the wiki.

    The locker is a bunch of web services that all natively speak REST and JSON to each other, all independent but connect to each other via one core web service, lockerd.js.

    The core service:

    • Installs services
    • Starts/stops any service
    • Manages how to access any service
    • Passes events (push) between services
    • Schedules upcoming task/job callbacks

    Individual services themselves are designed to be very isolated and loosely coupled:

    • They are started in their source code directory
    • They are provided a single persistent working directory and localhost port to listen on
    • Their working directory is for any stored state and data files
    • Any “session” information is passed via STDIN on startup from core
    • They should be be able to be turned off/on safely as needed
    • Any interface and controls for the locker owner are via html5 served on /
    • They always live behind a reverse proxy (core) that manages access
    • All access is via straight HTTP, as services within a locker may be spread out across devices, computers, and cloud instances

    Now, in order to know what services are providing what functionality and data formats, there is a concept called “service types” which provides a definition of:

    • One or more REST calls
    • The structure of the JSON data returned (attributes, values, etc)
    • Other data types returned (thumbnails, files)
    • What events are generated and their structure

    Examples of service types are like contact/twitter, contact/gmail, photo/flickr, status/facebook, etc.  Any service within the locker needs to define what types it provides, and can ask core how to reach or listen to other services based on their types.

    That’s pretty much it for the fundamental architecture in a nutshell!  That’s only part of the story though, very important is the common patterns that have been evolving around this architecture that most of the services are using in some form:

    • Most of the services are very event driven and written in node.js, and others in python (but as above, the arch is ultimately language independent)
    • The three general role patterns are services being either a:
      • Connector – to external data source
      • Collection – aggregation of common data types like photos, contacts, etc
      • App – functionality built to experience/manage data from Connectors/Collections
    • Some of the services with larger datasets are using Mongodb for lightweight indexing
    • Any service syncing/archiving data should store it as simple JSON on disk for future accessibility

    That’s where we’re at so far and I’m sure it will continue to adapt, grow, and mature as more and more data starts flowing in and ways to make it awesome spring up.  On the near horizon is also patterns involving TeleHash and interconnecting lockers, and making them easy and portable to be run anywhere.

     

  • quartzjer 2:03 pm on June 6, 2011 Permalink | Reply
    Tags: Trust   

    Tenets of Digital Trust 

    Since one of the most important long-term goals of the Locker Project is enabling a more powerful sharing platform between people (interconnecting Lockers via TeleHash), I’ve spent a lot of time thinking about the fundamentals of sharing and communicating online, and it always seems to come back to one intangible concept: Trust.  I’ve been breaking this into more traditional components to understand them better and wanted to share these ideas about the tenets of *digital* trust, which has real-life trust analogies but also more complexities being online.  I like to apply each one of these to any ditigal system to help guide my understanding of how it creates trust:

    • Authenticity
    • Verifiability
    • Security
    • Transparency
    • Consistency

    Authenticity is essentially the digital identifiers that are associated with someone and the confidence in whatever system generates those identifiers, that they represent the same person when repeated.

    Verifiability is the degree of your ability to establish the authenticity of someone, either actually in person or via another trusted person or system.  It typically precipitates and helps build authenticity, and comes into question when something unexpected or important happens.

    Security is the confidence in the integrity of the computing system both that you’re using, and that the other person is using.  There’s less trust when using public terminals or if suspicious dialogs happen on your own system, and equally so you wouldn’t share something important to a friend who’s using a possibly compromised system.

    Transparency is all about user interface and messaging, it’s how clear and consistent the tools and dialogs are in communicating what is happening.  It’s about creating complete expectations and delivering within those without trying to hide anything.

    Consistency is the complete experience over time and the most obvious one. Fundamentally, does the interface and do the identifiers create a predictable pattern that build confidence in someones digital experience.

    Since this is just a snapshot of my current thinking it’s likely to evolve and change as I try to apply it to more and more online experiences (like, is Security really worth highlighting on it’s own, or is it just a consequence of the others?).  For now it’s been a very handy way to tease apart some of the complexities of digital trust as we build the learnings into the foundation of the Locker Project, aiming to do a great job at connecting people with their data and better empowering them to share safely with others.

     

    • mushon 6:41 am on June 10, 2011 Permalink | Reply

      One more “tenet of *digital* trust” for you to consider:Confidentiality is the ability to make sure my shared data aimed at a specific social context is not compromised by leaking outside of that context. The malleability of digital data obviously makes this a hard task, but there are ways to create a culture of confidentiality (people don’t build Twitter robots to retweet private accounts, even though they can) and I believe that needs to be baked into the core of TLP.

    • messageTH 10:39 pm on June 28, 2011 Permalink | Reply

      Like this post. Thanks for sharing. How bout adding this to the elements needed for creating trust? Community is created when you consistently act to show that you have an interest in engaging in 2-way dialogue with me. This lessens my fear that you’ll take advantage of me and leave.

    • smurthas 12:03 am on June 29, 2011 Permalink | Reply

      Both are excellent points. Confidentiality is closely tied to Security and Consistency, but stands on its own as well. Community is hugely important and (among other things) acts to reinforce Verifiability, Transparency, and Consistency.

    • Fellow Traveler 10:45 am on July 4, 2011 Permalink | Reply

      Is the data inside my locker encrypted?Is it possible for Apps to have private data, that other Apps can’t access?How do I selectively give access to a 3rd party to SOME data, without giving him access to private data?Just wondering on the specifics of this.

  • quartzjer 4:04 pm on June 2, 2011 Permalink | Reply  

    The First of Many 

    I’ve been meaning to get a tech update out for a while now, but we’ve just been so busy!  I know, it’s a terrible excuse.  We’re an open source project and I feel we owe a bit more transparency.  That said, I’m going to do my best to try and make this a weekly update.

    Let’s get on to the news.

    Logistics

    A few up front logistics items.  We’ve moved the project to an organization page and repository.  We can now be found here.  You can either reclone your repository or change the origin url.  Jump on IRC if you are having issues with that.

    We are merging our previous dev branch into master permanently.  Too much of our work is hidden without some digging because we focus on the dev branch currently.  This has also led to confusion for testers as they run and then submit issues against the generally older master branch.  After the merge we’ll continue operations as normal, only spinning branches for larger changes and point releases.  Tags will be the primary method to track individual versions.

     

    Data Store

    The question of a backing data store for the system has come up many times, and we’ve finally settled on MongoDB.  All our core Connectors and Collections are starting to use it with great results.  Fear not for your simplicity though, one of the goals that we’ve had is to store data in a very basic and historical manner as well.  To do that Connectors maintain a simple disk based JSON file of all their synced activity.  This allows the Connector to easily reproduce current state if we need to change design goals, as well as provide you, the user, with a complete and simple backup forever.

    Another important point is that the use of Mongo does not preclude other data stores.  The Locker is an evolution with data and one pattern can not be used to reflect all data out there.  We will highly suggest that people try and use the existing Mongo, but if a new data set truly does not work in Mongo we’ll definitely help work out integration of other data stores.

     

    Query Interface

    Now that the data is accessible in a more common and powerful manner (read: Mongo), it’s natural to want to query into that data.  We’ve made a first pass implementation of a query API.  Most of the parts described in *Core Locker Data Access API* are implemented and available at the /query endpoint of the core.  This is an area we know needs growth and more iterations so I encourage you to play with it and come help us make it better.

    Dashboard

    The new dashboard look and feel is in from Justin Mecham.

    Screen_shot_2011-06-02_at_10
    Screen_shot_2011-06-02_at_10

    See the full gallery on Posterous

    This is definitely a step up and we’ll have a lot more coming in the user land soon.

    Wrapping Up

    All of the parts we talked about here, plus all the other work under the hood is being quickly wrapped up for a new release.  This is an important release because it has a more unified stack, as well as a more user friendly interface.  We’re still a bit more focussed on developers, but we’re getting really close to general use.

    We’re tightening up and moving faster, so keep watching for more updates!

     

     

    • jsmecham 4:10 pm on June 2, 2011 Permalink | Reply

    • ctide 5:18 pm on June 2, 2011 Permalink | Reply

    • Zooko 6:59 pm on June 2, 2011 Permalink | Reply

      “The question of a backing data store for the system has come up many times, and we’ve finally settled on MongoDB.”Is there a public record of the discussion?Regards,Zooko

    • Thomas Muldowney 9:35 pm on June 2, 2011 Permalink | Reply

      The conversation started at the earliest stages of the project and to be honest Mongo was in an early lead due to comfort from the core people that had used it. Most of the conversations were held verbally and not recorded. Later we decided to architect things so that we weren’t bound to a specific data store system, and I did a quick pass of testing basic operation of a few systems and we ended up looking at Mongo and Couch. Both seemed fine, but the comfort level was higher on the Mongo side. A few conversations are captured in the IRC logs, but the community is still getting active and the initial choice was made to use Mongo, at least for collections some of the core people are actively working on, with a door left wide open to doing others or expanding.

  • quartzjer 7:41 pm on May 16, 2011 Permalink | Reply  

    Brainstorming TLP Core Values 

    I’ve been wanting to share the core values that both inspired the founding of the Locker Project last year and that are driving me so passionately to advance it, but it’s been challenging to summarize such things.

    The first and most central value is that of personal data ownership.  That may not mean much to most people, but let me try an analogy: it’s been tried often in the past to create micro economies where you get paid by an employer in company “credits” that are useful only at the company store, you never really owned your paycheck/money.  I feel like that’s the world that we’ve fallen into when it comes to our personal data, and anything we can do to have more ownership over our data will free us to create a more vibrant digital economy.

    Now, it’s not enough to simply own (or at least have a copy of) your data, it has to be useful to you.  That’s the second core value, the ability to get immediate and direct personal value out of your data.  It’s yours, you should be the first and primary beneficiary of it, and you’ll need an open platform and great tools in order to do that.

    My third core value builds on the first two and is the very fabric and definition of (digital) freedom.  Everyone must have the capability to both share (or not share!) their data and the right to make those sharing decisions without bias.  This one is difficult to summarize but it’s central to any society, that individuals can connect independently and are free to decide to share things between them without any oversight or gatekeepers.

    My first attempt to simplify these into three statements a while back was this:

    • We believe you should have the ability to make a copy of your data
    • We believe your personal data should be always available and most useful to you
    • We believe that you should have have the freedom to share your data however you want

    They weren’t quite right, and the “We” felt like it created a separation, so I refined them further, here’s my current core values draft:

    • I own my personal data
    • I want my data to be useful to me
    • I make the decisions to protect or share my data

    Possibly a bit too far simplified, but they’re a work in progress. I wanted to openly share them with everyone for feedback as well as give some background on my personal drive in building this platform.

     

     

    • Devon 9:17 pm on May 16, 2011 Permalink | Reply

      This is my breakdown of your core values:I = Sovereign IDentity (I)I = Legal Structured Asset (OWN)Data = relational to I on per transaction basis (Data Value)I think you are on the mark in all regards.Its the proper definition of ‘I’ on universal Terms that is holding us back. Without it, ‘you’ and ‘me’ are distinctly un-relational, and data is managed in unpredictable ways with a loss of transactional efficiency.You can define and trust ‘I’ for you. But you can not service ‘I’ for me on those same Terms presently. Without I & I being systemically equal, their is loss in the system caused by the distortion which is being leveraged against I’s everywhere…all the time.

    • Miten Sampat 12:21 am on May 17, 2011 Permalink | Reply

      Thanks for starting this effort Jeremie, it is too important a problem not to try :-) the three pillars of your thinking are spot on. at the same time, in protecting ones data, it is important to realize the tremendous value each of the platforms give you (in terms of content, services, and personalization).the right answer here will ensure a balance between the need for services to leverage “aggregated data” generated by consumers to make their services better; and one that allows consumers a mechanism to transact specific data on a case by case basis and enforce their rights when they want.fascinating times. PS: i look forward to seeing you at PII 2011

    • Francis Fallon 11:47 am on May 17, 2011 Permalink | Reply

      The only thing that I would add is, and maybe this seems inherent already but, words are powerful-I want MY data to be MOST useful to MEor,I want to HARNESS my dataTo me at least, it’s more than just about ownership and control – it’s about ACTIVE empowering. Leveraging data should do more to benefit each of us than any one else. In my opinion, it’s about being in the drivers seat with my foot on the pedal and my hands at 10 and 2, not just owning the car or the keys.Jeremie, I am not a computer guy- just an advertising strategist very passionate about this cause.When the time comes that you need help getting word out there about this, I would love to champion this cause and join your team should you ever need a loud mouth.Thanks.

    • quartzjer 12:14 am on May 19, 2011 Permalink | Reply

      Thanks for the great feedback from everyone, it’s so exciting to know that many others feel the same urgency and dedication around this kind of model :)

  • quartzjer 11:56 pm on April 29, 2011 Permalink | Reply  

    Finally, a blog! 

    There’s been so much happening on the Locker Project lately that we’re way overdue for a community blog :)

    You’re going to find most of the heavy code activity at github and daily discussion around it on freenode in #lockerproject, with project announcements and updates happening on twitter, leaving this blog as a place for anyone working on this effort to expand and share their thoughts and create larger discussions around what we’re building.

    All of us have a big backlog of things we want to talk about here so in-between codeathons we’ll start posting some of that regularly.  If there’s anything you’d like to see just ping us via twitter/IRC, thanks!

    Jer

     

     

c
compose new post
j
next post/next comment
k
previous post/previous comment
r
reply
e
edit
o
show/hide comments
t
go to top
l
go to login
h
show/hide help
shift + esc
cancel
Follow

Get every new post delivered to your Inbox.