Digital Legacy
2022 | HCI Research | Creative Technologist
Instructor: Zach Pino | Team: Jorge Martinez Arana, Sandhini Ghodeshwar, Shinichiro Kuwahara, myself
IIT Institute of Design x NEC America
MEmiro:
Our data embeds our identity, our history, and our memory. Digital technology has created the possibility for data embedding experience to persist beyond a user’s lifetime. However, digital tools have made memories harder to hold. Considering the vulnerability associated with aging and cognitive decline, this project addresses the infrastructure of digital legacy and memory data.
The Digital Legacy project is a design approach to infrastructure digital legacy with greater intentionality and to provide tools for people reflect on the past. In this system, all data about the self can be a source of memories. Users can securely interact with their memory data to categorize, compress, synthesize, and distribute their legacy with their privacy and access preferences in mind.
Memiro is an approach to designing data-driven self-tracking experiences that are sensitive to vulnerability. This is one of four projects that applies this vision to a vulnerable scenario for individuals to develop trust in themselves in moments of hardship.
Project Background
Our team of researchers from the IIT Institute of Design focused on self-monitoring for the advancement of wellbeing, a broad topic with an enormous impact on the future of technology, design, and healthcare, sponsored by NEC America. The project, Memiro, or me miró, is Spanish for I look at myself, the reflective activity where we locate our research. The team consisted of Jorge Martinez Arana, Sandhini Ghodeshwar, Shinichiro Kuwahara, and myself, led by our advisor Zach Pino.
Our research aspired to identify technology’s role in advancing self-determination in personal and collective wellbeing, understanding data visualization and reflective practice on behavior change, prototyping with generative and computational processes to ensure inclusive outcomes, and exploring biodesign and biomaterial applications. We provided NEC with strategies, frameworks, and solution spaces to advance its technology capabilities toward wellbeing at the end-user and organizational levels.
Learn more on our microsite and on the Institute of Design website.
Locating Data in Experience
A Human-centered Competitive Landscape
Technology companies may not all be designed for user experiences but they can all be informed by them. Whether or not the product, service, or solution reaches an end-user or an intermediary, it is important to understand the layers in which these data-oriented services are located.
The competitive landscape for data products ranges from wearable devices, entertainment algorithms, DNA heritage, and analytical cloud services. Despite the diversity of these applications, they can be filtered through four areas of user experience: Intent, Affordance, Context, and Analysis.
Filters of Experience:
Intent: Self data, originating from user directed activities
Affordance: Data which arrives based on possible actions or behaviors from the user
Context: Data produced adjacent to user activities and environments or afforded technology
Analysis: Data resulting from gathered data sources relating to the user indirectly
Data Governance Chain
Data driven systems have clear value for agile organizations. But without a clear understanding of the data generating sources, the role of the data holder, and the filters that lie between, organizations lack holistic impact.
Human-centered design (HCD) methods focuses on the end-user in order to involve them and their experience in the development process. Data driven organizations can benefit from this methodology by focusing on individual user data to benefit collective understanding. Data governance is required to ensure the flow of information is robust, ethical, and impactful.
Memiro repositions self-tracking and data-collection organizations as data custodians in terms of the responsibility they hold to data generators, with the definitions below:
Data Generator: An information source where data is generated passively, actively, automatically, or manually. May include the human body, the environment, and other measurable self-generative processes
Data Custodian: An entity which governs the flow of information, manages data processes, and protects the privacy of data generators
Designing with time
Needs change throughout the life of users, businesses, and communities. Most existing forms of self tracking focus on a specific concern bounded by time. Others are generalized across multiple experiences, but are often disconnected from the user's identity. Systems that are cognizant of the passage of time are able to use data more effectively. These solutions bring focus to transition and can adapt to long term change.
Strategies which design for change embed an awareness of the possibilities and limitations of data capture. As people and organizations age, considering how a user experience and its data change throughout a user lifecycle is critical to advancing innovation.
Perhaps age-specific technologies can adapt once the user grows out of the product, or services can provide assistance when people are at both their peak and decline of their health. These solutions empathize and support change rather than show judgement.
Involving the Self
As the gap between digital and virtual diminishes, as a culture, we have seen an increasing need for unplugging and reconnecting with the self. Most self-tracking technologies which focus on improving self-efficacy rely on passively collecting, analyzing, and applying information collected from biomarkers. Moreover, the data collected is personal to an individual, yet minimal steps are taken to prevent data misuse. Considering user vulnerability, regardless of the distance of the user to the product, is paramount to creating value in a fearful culture.
In such a situation, it becomes critical to focus on actively involving the self in generation, analysis, and interpretation of personal data, which could be done via the individual building their own data literacy. Since self-tracking deals with personal data, it’s imperative to take ethically responsible decisions to secure sensitive information. This could be done via providing ownership of data to the individual and decentralization of data sources, such as using technologies like edge computing or blockchain.
Ecologies of Scale
Collecting individual data from the bottom-up helps organizations improve diversity and inclusivity and broaden the population of products, services, and systems; while top-down approaches in the market mainly target normalized or mainstream people and may ignore marginalized voices.
In encouraging users to share their data tracks, user agreement on data sharing is a key to scale up data-driven solutions. Designing persuasive, accessible, and user-friendly experiences is important to nudge people to share their data both for individual and collective benefit.
Organizations must set boundaries to improve data privacy, data accessibility, and data ownership in order to build trustworthy and well-balanced systems that can know the value as well as the risks of collective data. For example, a subset of users (individuals, families) might access private data while others are limited to filtered data to mitigate risks of data leakage. Regarding data ownership, a system could allow individuals to delete or revoke their data across the system. Similarly, community members could agree on a requirement to delete data for the community. Avoiding polarized attitudes on those sensitive issues gives an opportunity to achieve system optimization without exploitation.
Collective data has a potential to provide value where individual data cannot. In a smart city context, city governments may build customizable infrastructures in which service providers can offer consistent services on top of infrastructure for specific user groups throughout their lifetime. Newly generated risks should be taken into consideration for different time frames, user groups, and data applications.
Context
When lives are digital, how are we remembered?
With contemporary electronic devices, people could theoretically remember anything with the click of a button. But the reality is: relying on digital tools has made memories harder to hold. Memory recall becomes even more of an issue for people experiencing cognitive decline or dementia. The ability for people to share their lives and identities through digital technology has created the possibility for experience to persist beyond the lifetime of the user. With populations aging worldwide and digital amnesia on the rise, design must consider how digital experience supports memory making, legacy generation, and data distribution.
Research through Design
3rd Person Experience: IoT and Smart Home Tech
Assembling existing technology for new applications raised questions of how systems might adapt to memory documentation. The concept was initially constrained by a scenario in which a user suffers memory loss and wants to capture daily experiences for themselves or their loved ones. With the belief that identity is embedded in daily actions and behaviors, passive memory collection was explored through smart home technology and automation. Through integration of Apple HomeKit, Apple Shortcuts, Philips Hue bridge and lights, an Arlo wireless smart home camera, a generic motion sensor and an Aura digital picture frame, the prototype constructed a space in which memories could be collected and displayed automatically when triggered by physical presence. The result was a third-person perspective of one’s experience, where images reveal motion and activity different from a composed picture in which the user is aware of the person on the other end of the camera.
1st Person Experience: Wearables and Analog Media
The second iteration of passive collection for memory was through wearing a body camera. Body cameras, or body cams, are often associated with accountability policies for law enforcement officers, or GoPro-style action videos. In this experiment I wore a body cam while performing everyday activities such as driving, grocery shopping, and walking around my home. Despite brief clips of my face in a mirror, the outcome resulted in video clips of context without identity, primarily unfocused movement which made sense only to the wearer. A few stills from the body cam were used to create cyanotype prints to remove color information (other than blue) and convey the image as a moment for remembering rather than observing as part of
the present.
Legacy Infrastructure
The previous design approach surfaced team discussions regarding user experience, data modeling, and data privacy. In order to understand each problematic area, the digital legacy infrastructure as a whole was diagrammed to clarify the minimum interactions between user, data, and time domains. The infrastructure is composed of five key phases: Memory Capture, Memory Recollection, Artifact Sharing/Inheritance, Legacy Compression, and Legacy Externalization. These phases occur on two time-bound layers: Conscious Experience — which encompass user decisions and preferences, and Legacy System — the computational processes applied to data on the back end.
The first phase is Memory Capture which encompasses both active and passive collection of memory data. From the collected information, the memory object will be composed of information about the user, the technology and context of capture, the media type and details, the context of the memory including location or recognized content, sentiment based on information derived from the data and user input, and how the memory correlates to other memory objects.
Diagramming Memory Over Time
The next phase of Memory Recollection is the process by which memories are periodically resurfaced. The process causes the user to recollect that specific memory and adjust its significance, recollection frequency, and classification. Through the process of memory recollection, memory objects will update such that the data reflects whether they are deemed more significant or more valuable, thereby appearing more or less in a legacy artifact. The legacy artifacts are composed of several memory objects that represent an aggregate experience.
Following Memory Recollection is the Artifact Sharing and Inheritance phase. The system should support the ability to share memories and legacy artifacts just as people share photos. Legacy artifacts describe an experience through several memories, like a collage or a short animation.
Legacy Externalization is the final phase in which legacy is distributed to external parties such as healthcare providers to support medical research, estate planning to distribute assets among beneficiaries, organizations to commemorate culture or archive institutional knowledge, and the shared network of loved ones who inherit intimate information. The legacy infrastructure must consider the type of information and security access based on user preferences and how those may change after death.
The phase that is working in the background throughout the user’s lifetime is Legacy Compression. When memories are surfaced during the recollection phase, the cumulative legacy becomes crystallized through the compression of the multiple memory objects into one. In other words, what might be a digital equivalent of a personal time capsule? What objects, experiences, and information communicates the whole of a life?
What might memory data look like?
Materializing Memory
Prototypes of legacy artifacts from the system were produced to shape the system’s design language and suggest how the phases of the legacy infrastructure may materialize.
Through the prototyping process DALL·E and images under creative common license were used to represent relatable and realistic, yet anonymous depictions of memorable events.
DALL·E — “a middle aged couple looking at each other across a pale green table in the style of pointillism with ginkgo leaves”
As a design language, the Ginkgo was chosen as a visual representation of memory both for its reputation as a memory-enhancing supplement as well as Ginkgo biloba being an ancient tree species. The irregular yet recognizable outer edge of the ginkgo leaf would embody a data visualization of the user’s engagement with recollected memories. The radial veins of the leaf encompass the various data components of a memory. The face of the leaf can serve as a projection surface to capture the visual impression of a memory.
Using the visual language of a ginkgo tree, a ginkgo branch could be a living projection in the home to convey the growth and pruning of a user’s legacy where leaves are data visualizations of individual memories. The intent of a projection is to bring data visualization in a living space, similar to how a child’s height is marked on a door jamb year over year.