Microsoft Soundbyte
tools :
Figma, Invision, Photoshop, Illustrator, Miro
Microsoft Soundbyte earbuds detect and filter harmful sounds, addressing noise pollution and restoring acoustic quality through a dedicated app.
Our challenge was to seamlessly integrate individuals with hearing loss into workplace communications, regardless of location, addressing missed audible cues or announcements.
As a UX Strategist and Art Director, I led the product's conceptualization and user-centric iterations in partnership with Microsoft engineers. While it didn't reach the market, our collaboration resulted in an innovative prototype showcasing its future potential. This project was recognized by Creative Conscience as a finalist for best social impact and received the Gold award at the York Festivals of Advertising for Positive World Impact.
PRODUCT INNOVATION
In collaboration with Tandon School of Engineering students in NY, Microsoft is revolutionizing smart earbuds for the 460 million people worldwide with uncorrectable hearing loss. These earbuds use cutting-edge technology, prioritizing workplace safety in noisy environments and offering preventative ear care.
DESIGN PROCESS
In developing Microsoft Soundbyte, we combined the creation of a mobile application with a wearable device, guided by Inclusive Design principles. This approach emphasized understanding diverse user needs, particularly those with hearing loss. We iteratively refined our prototypes through extensive user testing, ensuring our solutions were both inclusive and intuitive. To document our journey, we compiled a 140-page booklet, providing a comprehensive record of our process and a complete product introduction, showcasing our commitment to a user-centered and inclusive design approach.
POTENTIAL SCENARIOS
Our product is designed for people regularly exposed to loud environments like construction sites, airports, and bars. They rely on cumbersome earmuffs or small earplugs, which often impede clear communication and timely information.
USER RESEARCH
Kelly Tremblay, Ph.D., an expert in auditory neuroscience and a professor in the Department of Speech & Hearing Sciences at the University of Washington, who is also a member of the HLAA Board, talked about the brain's involvement in auditory processing and the way auditory rehabilitation can enhance this brain function.
Interview with people with hearing-loss
Numerous individuals are self-conscious about wearing hearing aids. While these devices are beneficial in typical situations, they can become bothersome in environments with echoing or excessive background noise.
Interview outdoor workers
Construction workers have noted that walkie-talkies, noise-canceling headphones, and earbuds are crucial for daily work. However, these devices often hinder their ability to hear alarms, posing a risk of hazardous incidents.
Microsoft Seeing AI Team and Soundscape Team
We held Skype interviews with Microsoft's Seeing AI and Soundscape teams, gathering insights on user testing and design for people with disabilities. Their input, especially on sensory substitution from the Hearing AI team, guided our approach to innovatively assist individuals with hearing impairments, focusing on visualizing sound beyond traditional auditory methods.
IDEATION - BRAINSTORMING 1.0
We refined our understanding of our target user group by discussing possible scenarios, analyzing a spectrum of personas, and conducting contextual inquiries. This enhanced understanding significantly informed the design of our product. Starting with the concept of intelligent noise-cancelling earbuds, we expanded our thinking to concentrate on distinct features that our product should encompass.
COMPARATIVE ANALYSIS
We conducted thorough market research, immersing ourselves in the competitive landscape and exploring advanced technologies. This extensive analysis provided vital insights, guiding our strategic alignment with industry trends. Our findings revealed that while current smart earbuds emphasize versatility and portability, they often overlook sound and hearing – the core aspects of earbuds. This suggests a misdirection in the industry's focus. Additionally, we noted that existing apps have limited functionalities, primarily centered on health tracking, AI, and basic settings. Recognizing this, we saw an opportunity to develop new functions tailored to the various scenarios our users encounter daily.
Another issue with current applications is the lack of clear differentiation between the functionalities of wearable devices like earbuds and those for wristbands or headbands. Apps designed for earbuds must prioritize different features, such as filtering out ambient noise while enhancing important sounds, rather than simply replicating the functions suited for other wearables.
MIND MAP
In our research on intelligent earphones and wearable apps, we identified sound generation and AI assistant integration as key features for our earbuds. These earbuds are designed to filter out background noise and adjust frequencies to enhance communication. During a two-week mind-mapping session, we delved into treatments, applications, and environmental noise impacts, such as construction sites and subways. We also explored the integration of AI assistants and the role of earphones in sound detection, ensuring our approach was comprehensive and well-informed.
ADVANCED AUDIO TECHNOLOGIES IN FOCUS
Sound Quality and Timbre Analysis
Understanding timbre is crucial for differentiating sounds with similar pitches and loudness, like distinguishing a piano note from a clarinet. This characteristic, essential for voice recognition, depends on the sound wave's shape, influenced by fundamental sounds and their harmonics. Timbre’s determination is linked to the sound source’s physical properties and oscillation method.
Speaker Diarisation Techniques
Speaker diarisation, answering 'Who spoke when?', combines speaker segmentation and clustering. It partitions audio streams into homogeneous segments to identify speaker changes and assigns these to specific sources, enhancing audio document readability and aiding in speaker identification with recognition systems.
Sound Detection and Recognition
Our software utilizes a database of familiar sounds (alarms, voices, etc.) stored on the phone. It identifies sounds by matching them with database entries, smoothing high-frequency or dangerous sounds. It also associates sound features (frequency, timbre, amplitude) with sources, like 1-3 kHz alarm sounds, helping reduce noise-induced health risks.
Active Noise Cancellation
Noise-cancelling headphones employ active noise control, reducing ambient sound by introducing a sound wave designed to counteract the original.
Acoustic Positioning Methods
Acoustic location, developed in the 1940s, employs various methods to identify and position sounds. Techniques range from perceptual-based 3D sound localization, effective in specific listening areas, to the Doppler effect, used in devices for detecting sound positions.
Exploring Bone Conduction Technology
Most sounds are heard through earbones and eardrums. Research on bone conduction is vital for hearing loss types where the eardrum or ossicles are compromised. Bone-conduction devices bypass the eardrum, converting sound waves into vibrations the cochlea receives, presenting a promising alternative for those with specific hearing impairments.
SITEMAP
Our Application's Sitemap presents a clear, structured overview of the app, detailing the layout and connections between different sections. This visual guide is crucial for aligning development and user experience goals, highlighting critical areas like the home page, settings, sound customization, and support. It ensures a streamlined design process and an intuitive user interface.
EVOLUTION OF HAND-SKETCHED PROTOTYPES
Our initial foray into prototyping began with Paper Prototype 1.0, where we laid out fundamental features such as earbud pairing, AI assistant integration, and three primary functions: noise filtering, friend contact, and sound frequency adjustment to suit individual hearing ranges.
Progressing to Paper Prototype 2.0, we refined our initial design based on feedback from tests conducted with friends and team members. This iterative process led to a more logical flow in user interactions. The upgraded prototype introduced a novel function that allows users to mute sounds from specific directions on a map. Additionally, we incorporated equalizers on the filter page, giving users direct control over adjusting various surrounding sound sources.
USER TESTING
Our initial user testing session provided valuable insights. While users experienced some navigation confusion and uncertainty about the app's functionalities without our guidance, this feedback was instrumental in highlighting areas for improvement. The testing underscored the importance of intuitive design and clear user instructions, allowing us to enhance the app’s user experience. This constructive input is a vital step in our iterative design process, helping us to refine the application to be more user-friendly and self-explanatory.
LO-FI PROTOTYPE
Responding to the insights gained from our initial user testing, we meticulously revised our interface, enhancing button layouts and incorporating an instructional guide before the login page, along with detailed enhancements on subsequent pages. This iteration of the prototype concentrated on both the functionality and the user interface of our application. We conducted further user tests with a diverse range of our target audience. The outcomes of these tests were enlightening, providing us with unexpected and valuable perspectives that significantly informed our design approach, demonstrating the dynamic nature of user experience design.
Lo-fi prototype. Try on by click this.
USER TESTING: JOURNEY MAP INTEGRATION
For our second user testing phase, we incorporated a journey map, a critical step in refining our application. This map provided a clear visual representation of how users interacted with the app, from the initial point of contact to the execution of key functions. It helped us effectively identify and address critical touchpoints and any user challenges. The feedback from this testing was instrumental, guiding us to make focused improvements in the app's user interface and functional aspects. By scrutinizing the user's journey, we enhanced the overall usability and user experience in our app's development.
HI-FI PROTOTYPE
PRODUCT PROTOTYPING
Our product design journey began with a low-fidelity prototype, progressing to a detailed 3D model. Engaging in numerous collaborative sessions with stakeholders, including Microsoft engineers and prospective users, we actively solicited and incorporated feedback, refining our design at each step. This iterative process culminated in the 3D printing of a high-fidelity prototype, vividly demonstrating the advancements achieved through our comprehensive and responsive design approach.
Product design sketch
PROTOTYPES
Considering the insights gathered from the feedback on our second prototype, we strategically repositioned the bone conduction component to the back of the ear. This adjustment was a calculated move to enhance user comfort and the device's overall effectiveness.
3D Printed Prototype
Hand made Prototype
CAD 3D Prototype V2
3D Printed Prototype V2
CAD 3D Prototype
